[BUG]: RuntimeError: Found no NVIDIA driver on your system

[IMParesqui]

Hello, I mounted the docker container based on your image, made a few changes and then comited the new image.
When I try to run on runpods I get the following error log:

2025-01-29T14:18:35.270476069Z ==========
2025-01-29T14:18:35.270485659Z == CUDA ==
2025-01-29T14:18:35.270524749Z ==========
2025-01-29T14:18:35.273021706Z CUDA Version 11.8.0
2025-01-29T14:18:35.273668766Z Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2025-01-29T14:18:35.274100205Z This container image and its contents are governed by the NVIDIA Deep Learning Container License.
2025-01-29T14:18:35.274101705Z By pulling and using the container, you accept the terms and conditions of this license:
2025-01-29T14:18:35.274105005Z https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
2025-01-29T14:18:35.274107485Z A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.
2025-01-29T14:18:35.281988516Z Symlinking files from Network Volume
2025-01-29T14:18:35.282930995Z ln: failed to create symbolic link '/workspace/runpod-volume': File exists
2025-01-29T14:18:35.284336403Z runpod-worker-comfy: Starting ComfyUI
2025-01-29T14:18:35.284412533Z runpod-worker-comfy: Starting RunPod Handler
2025-01-29T14:18:35.319565344Z [START] Security scan
2025-01-29T14:18:36.058164141Z [DONE] Security scan
2025-01-29T14:18:36.094298690Z ## ComfyUI-Manager: installing dependencies done.
2025-01-29T14:18:36.094314640Z ** ComfyUI startup time: 2025-01-29 14:18:36.094255
2025-01-29T14:18:36.094317720Z ** Platform: Linux
2025-01-29T14:18:36.094336090Z ** Python version: 3.10.12 (main, Nov 6 2024, 20:22:13) [GCC 11.4.0]
2025-01-29T14:18:36.094353890Z ** Python executable: /usr/bin/python3
2025-01-29T14:18:36.094383510Z ** ComfyUI Path: /comfyui
2025-01-29T14:18:36.094407590Z ** Log path: /comfyui.log
2025-01-29T14:18:36.946724540Z Prestartup times for custom nodes:
2025-01-29T14:18:36.946729330Z 0.0 seconds: /comfyui/custom_nodes/rgthree-comfy
2025-01-29T14:18:36.946754180Z 0.0 seconds: /comfyui/custom_nodes/ComfyUI-Easy-Use
2025-01-29T14:18:36.946797680Z 1.6 seconds: /comfyui/custom_nodes/ComfyUI-Manager
2025-01-29T14:18:36.964775470Z --- Starting Serverless Worker | Version 1.7.5 ---
2025-01-29T14:18:36.964927520Z {"requestId": null, "message": "Starting API server.", "level": "INFO"}
2025-01-29T14:18:36.995001506Z INFO: Started server process [46]
2025-01-29T14:18:36.995072746Z INFO: Waiting for application startup.
2025-01-29T14:18:36.995230846Z INFO: Application startup complete.
2025-01-29T14:18:36.995600825Z INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
2025-01-29T14:18:37.802486306Z Traceback (most recent call last):
2025-01-29T14:18:37.802497866Z File "/comfyui/main.py", line 90, in
2025-01-29T14:18:37.802538306Z import execution
2025-01-29T14:18:37.802547676Z File "/comfyui/execution.py", line 13, in
2025-01-29T14:18:37.802587196Z import nodes
2025-01-29T14:18:37.802600106Z File "/comfyui/nodes.py", line 21, in
2025-01-29T14:18:37.802641585Z import comfy.diffusers_load
2025-01-29T14:18:37.802648595Z File "/comfyui/comfy/diffusers_load.py", line 3, in
2025-01-29T14:18:37.802687855Z import comfy.sd
2025-01-29T14:18:37.802695405Z File "/comfyui/comfy/sd.py", line 5, in
2025-01-29T14:18:37.802733845Z from comfy import model_management
2025-01-29T14:18:37.802749345Z File "/comfyui/comfy/model_management.py", line 143, in
2025-01-29T14:18:37.802790135Z total_vram = get_total_memory(get_torch_device()) / (1024 * 1024)
2025-01-29T14:18:37.802803975Z File "/comfyui/comfy/model_management.py", line 112, in get_torch_device
2025-01-29T14:18:37.802849885Z return torch.device(torch.cuda.current_device())
2025-01-29T14:18:37.802867535Z File "/usr/local/lib/python3.10/dist-packages/torch/cuda/init.py", line 940, in current_device
2025-01-29T14:18:37.802963615Z _lazy_init()
2025-01-29T14:18:37.802975445Z File "/usr/local/lib/python3.10/dist-packages/torch/cuda/init.py", line 319, in _lazy_init
2025-01-29T14:18:37.803031535Z torch._C._cuda_init()
2025-01-29T14:18:37.803062855Z RuntimeError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx

Any ideas why? I tried it several times, it works localy, but not on the pods...

[youh127]

Hi, has anyone managed to resolve this issue? I'm experiencing the same problem and would appreciate any updates or workarounds.

[TimPietrusky]

@IMParesqui @Younes127 sorry for the super late reply.

this seems very odd, would any of you please try the latest version of the image (currently 5.1.0) and tell me if you still run into this problem? i have never seen this when deployed on RunPod, but would love to get this solved

[TimPietruskyRunPod]

Status update: there are two recent changes that should address this:

1. fix: poll ComfyUI server indefinitely while process is alive #197 (released 5.8.3) — adds an explicit GPU pre-flight check that fails fast with a clear error instead of silently returning Found no NVIDIA driver.
2. fix(hub): broaden allowedCudaVersions and lower default log level #222 (under review) — broadens `allowedCudaVersions` in the hub config from `["12.7","12.6"]` to include 12.8/12.9/13.0 as well. The narrow allow-list meant the hub was scheduling workers onto hosts that didn't meet the driver requirement, and the OCI prestart hook from `nvidia-container-cli` was killing the container before the worker code could even run.

If you're still hitting this on a recent version (5.8.5+) once #222 lands, please reopen with the full container start log. The actual error message should be clearer now.

Closing speculatively — happy to reopen if you can still reproduce after the next release.