feat(vgpu): support the vendor-specific VFIO framework on Ada/Hopper+ GPUs - #195
Closed
lexfrei wants to merge 14 commits into
Closed
feat(vgpu): support the vendor-specific VFIO framework on Ada/Hopper+ GPUs#195lexfrei wants to merge 14 commits into
lexfrei wants to merge 14 commits into
Conversation
lexfrei
marked this pull request as ready for review
July 2, 2026 00:48
This was referenced Jul 7, 2026
lexfrei
force-pushed
the
feat/vendor-vfio-backend
branch
from
July 20, 2026 00:43
ef37506 to
7b4dd0f
Compare
Move the ME/NOME/MEALL/GFX suffix-stripping helper into the types package as StripAttributeSuffix so it can be shared by multiple vGPU config backends. The existing pkg/vgpu helper now delegates to it, keeping behavior identical. Signed-off-by: Aleksei Sviridkin <f@lex.la>
Since vGPU 17.0, the NVIDIA vGPU Manager driver on Ada, Hopper and newer GPUs exposes vGPU devices through a vendor-specific VFIO framework instead of the mediated device (mdev) framework. On these systems there is no /sys/class/mdev_bus: each GPU is an SR-IOV physical function bound to the 'nvidia' driver, and vGPU devices are created on its virtual functions by writing a numeric vGPU type ID to the per-VF 'nvidia/current_vgpu_type' sysfs file. Add a vfio package implementing the same Get/Set/Clear vGPU config operations as the existing mdev-based manager on top of this sysfs interface: - GPUs and their VFs are discovered by walking the PCI devices tree, keeping the same GPU index semantics as the mdev backend. - SR-IOV VFs are enabled on demand via the sriov-manage script shipped with the vGPU Manager driver, optionally through chroot into a host root mount when running in a container. - The set of creatable types is read from the per-VF 'creatable_vgpu_types' files. Because those files only list types creatable at the time of reading (allocated capacity disappears from them), an NVML-backed resolver provides the allocation-independent ID-to-name mapping needed to report currently applied configurations. - MIG attribute suffixes (ME, NOME, MEALL, GFX) are stripped from requested type names when only the base type is supported, matching the mdev backend behavior. The sysfs root, SR-IOV enable function and type resolver are all injectable, allowing the full apply/assert/clear flow to be unit tested against a fake sysfs tree. Signed-off-by: Aleksei Sviridkin <f@lex.la>
Select the vGPU config backend based on what the host driver exposes: a populated /sys/class/mdev_bus selects the existing mdev backend, while NVIDIA GPUs exposed as SR-IOV physical functions bound to the 'nvidia' driver (and no mdev bus) select the vendor-specific VFIO backend. When neither is detected, the mdev backend is kept as the default to preserve the historical behavior and error messages. The apply and assert commands use the detected backend transparently, so the same configuration file and CLI semantics work on both frameworks. The apply command gains a 'host-root-mount' flag, which the Kubernetes wrapper propagates to its child processes, so that sriov-manage can be run from the host driver installation when the manager runs inside a container. Signed-off-by: Aleksei Sviridkin <f@lex.la>
Describe the two vGPU device management frameworks, the automatic framework detection, and the operational specifics of the vendor-specific VFIO path: automatic SR-IOV VF enablement, MIG prerequisites for MIG-backed types, the NVML dependency for resolving already-created vGPU devices, and the boot-time vs live reconfiguration caveats. Signed-off-by: Aleksei Sviridkin <f@lex.la>
On GPUs whose vGPU devices are managed through the mdev framework (e.g. Ampere), the mdev parent devices are also SR-IOV virtual functions. Before the VFs are enabled, such a system is indistinguishable from a vendor-VFIO one, so the vfio backend may be selected for it. The VFs it then finds carry an 'mdev_supported_types' directory instead of the vendor-specific 'nvidia' directory, and the previous error message misleadingly blamed the driver installation. Detect this case and point at the mdev framework instead; a subsequent run selects the mdev backend, since enabling the VFs populates the mdev bus. SetVGPUConfig now validates all VFs before deleting any existing vGPU devices, so no state is modified through a wrongly selected backend. Also extract the sriov-manage argument construction so both the direct and chroot invocation forms are covered by tests along with the PCI address validation, and drop the unused gpu_instance_id accessor. Signed-off-by: Aleksei Sviridkin <f@lex.la>
Exercise all paths of the NVML-backed type resolver: successful resolution, already-initialized NVML, and the initialization, device handle, supported-types and type-name failure modes. The numeric type ID extraction is simplified to the unsigned-integer case (the only one the concrete go-nvml type can be) and covered for both the named integer and opaque struct cases. This makes go-nvml a direct dependency of the module. Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace the duplicated string literal with an exported constant in the apply package so the Kubernetes wrapper and the apply command cannot drift apart on the variable name. Signed-off-by: Aleksei Sviridkin <f@lex.la>
Automatic SR-IOV VF enablement relies on the sriov-manage script from a host-installed vGPU Manager driver; with a containerized driver the VFs are expected to be enabled by the driver container itself. Framework detection selects a single backend per node, so nodes mixing GPUs from both framework generations are not supported. Signed-off-by: Aleksei Sviridkin <f@lex.la>
Move the full suffix-stripping test table into the types package, where the function now lives, so its contract stays covered even if the pkg/vgpu wrapper is ever removed. The wrapper keeps a small delegation test. Also fix a field name typo in the vfio test fixtures. Signed-off-by: Aleksei Sviridkin <f@lex.la>
GetVGPUConfig reports vGPU type names as the driver knows them, while a config file may use names with a MIG attribute suffix (e.g. DC-1-24QGFX on boards where the driver only knows DC-1-24Q). A strict comparison between the two never matches, so a suffixed config was torn down and recreated on every reconcile even when already applied - including the GPU-operand shutdown and MIG reconfiguration cycle in the Kubernetes flow. Expose the suffix normalization both backends already perform during apply as Manager.NormalizeVGPUConfig, and compare the current state against the normalized config in the apply and assert commands. When normalization is not possible (e.g. the driver is not loaded yet), the comparison falls back to the raw config, preserving the previous behavior. While unifying the normalization logic across the backends, two related defects are fixed and covered by tests: - Config entries that collapse to the same supported type after suffix stripping were silently merged: the mdev backend kept only one of the counts (map iteration order decided which), while the vfio backend summed them. Both backends now reject such configs explicitly. - The vfio type lookup preferred a stripped name from sysfs over the exact suffixed name known to NVML, silently substituting the base type for the requested one. The exact name is now exhausted against all sources before the stripped name is considered. Signed-off-by: Aleksei Sviridkin <f@lex.la>
Explain how nvidia-vgpu-dm apply locates the sriov-manage script from the host driver installation when running inside a container. Signed-off-by: Aleksei Sviridkin <f@lex.la>
gpuAddresses relied on os.ReadDir returning sysfs entries in lexicographic order, which happens to match ascending PCI address order only for fixed-width addresses. go-nvlib's nvpci.GetAllDevices, which the apply/assert commands use to derive the GPU index passed to this package's Manager, does not make that assumption either: it sorts explicitly by parsing each address as a hex number. Apply the same explicit numeric sort here, so a GPU index always refers to the same physical GPU in both packages regardless of how sysfs happens to enumerate its entries. A wrong match here would mean SetVGPUConfig applies or tears down vGPU devices on the wrong physical GPU. Signed-off-by: Aleksei Sviridkin <f@lex.la>
A negative count in a VGPUConfig entry fell through the zero check that stops creating vGPU devices, so it never reached zero and filled every free virtual function with the requested type. Stop on any non-positive count so such a request creates nothing. The path is unreachable through the CLI, but the vfio package is exported. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
Enabling SR-IOV virtual functions shelled out to sriov-manage with no timeout and dropped the command's stderr on failure, so a hung script could block indefinitely and errors were undiagnosable. Run it under a bounded context timeout and include the captured stderr in the returned error. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
lexfrei
force-pushed
the
feat/vendor-vfio-backend
branch
from
August 10, 2026 08:55
a4388f2 to
4d31dc6
Compare
Member
|
Hi @lexfrei! Thanks for the contribution. We previously had a very similar PR (#139). However, when we actually tried launching a KubeVirt VM with a vGPU backed by the vendor-specific VFIO framework, we realized there were some gaps on the KubeVirt side. I think the scope of this change will require that the contributor is able to test this end-to-end since the team won't have bandwidth for back and forth testing with any changes required. That being said, if you're able to test this, we can re-open this PR and continue to discuss. |
Author
|
@karthikvetrivel We did test this end-to-end. This branch was the device-creation half of a pilot on an 8x H200 SXM (NVSwitch) host: around 40 vGPU devices across dozens of VMs, whole-card and MIG-backed (1g/3g) profiles mixed, licensed guests, running for weeks under KubeVirt v1.9.0-beta.0. Field report with the full stack: kubevirt/kubevirt#16890 (comment) About the KubeVirt-side gaps: I think #139 hit KubeVirt before 1.9. virt-handler had no way to pass a pre-configured vGPU VF to a guest. kubevirt/kubevirt#16890 fixed exactly that (VFs bound to the nvidia driver with a configured profile are picked up, PFs are skipped) and shipped in v1.9.0 on Jul 30. The rest of the integration lives in NVIDIA/kubevirt-gpu-device-plugin#192 (per-profile resource names) and NVIDIA/kubevirt-gpu-device-plugin#193 (NVLink fabric partition activation for whole-card vGPU on NVSwitch, the cuInit error 802 case). With those in place we didn't hit any KubeVirt-side gaps: VMs get their VFs, CUDA works, MIG-backed and whole-card coexist on one node. One honest caveat on test capability: that H200 environment is decommissioned, so I can't re-run there today. Everything above was validated against this branch while it ran. If review changes need re-testing I'll do it when I have vGPU-capable hardware again, but I can't promise a fast loop right now. I think this covers the end-to-end requirement. Can we re-open? |
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Since vGPU 17.0, the NVIDIA vGPU Manager driver on Ada, Hopper and newer GPUs (L40S, H100, H200, ...) exposes vGPU devices through a vendor-specific VFIO framework instead of the mediated device (mdev) framework. On these systems there is no
/sys/class/mdev_bus, so the current mdev-only implementation fails withno parent devices found for GPU at index '0'and vGPU devices cannot be configured at all. NVIDIA/gpu-operator#2594 describes the resulting gap in the GPU Operator vGPU stack.The consumption side is already in place: KubeVirt v1.9 (kubevirt/kubevirt#16890) discovers SR-IOV VFs bound to the
nvidiadriver whosenvidia/current_vgpu_typeis non-zero and advertises them as PCI host devices. What is missing is the piece that creates the vGPU devices on the VFs — which is exactly this manager's job.What this PR does
Adds a second vGPU config backend implementing the same
apply/assert/clear semantics on top of the vendor-specific VFIO sysfs interface, and selects the backend automatically:/sys/class/mdev_busselects the existing mdev backend (unchanged).nvidiadriver (and no mdev bus) select the new vendor-VFIO backend.The same configuration file format, CLI flags and node label semantics work on both frameworks.
The vendor-VFIO backend:
/sys/bus/pci/devices, keeping the same GPU index semantics as the mdev backend.sriov-managescript shipped with the vGPU Manager driver (throughchrootinto the host root mount when running containerized, following the same pattern the Kubernetes wrapper already uses forsystemctl).nvidia/current_vgpu_typesysfs file, after checking the per-VFnvidia/creatable_vgpu_typeslist. Writing0deletes a vGPU device.nvmlDeviceGetSupportedVgpus+nvmlVgpuTypeGetName), because thecreatable_vgpu_typesfiles only list types that can still be created: once existing vGPU devices consume the GPU's capacity, the applied types disappear from them, andassert/idempotency checks would otherwise be impossible.MIG-backed vGPU types reuse the existing mig-parted flow in the Kubernetes wrapper unchanged: MIG mode is enabled and GPU instances are created before the vGPU types are applied, and the driver then binds each vGPU device to a free GPU instance.
Testing
The sysfs root, SR-IOV enable function and NVML type resolver are injectable, so the new backend is unit tested end to end against a fake sysfs tree: discovery/enumeration,
creatable_vgpu_typesparsing, whole-card and MIG-backed apply, clear, re-apply idempotency, capacity exhaustion, unsupported types (validated before any existing devices are touched), VF enablement, and the "attributes unreadable before a type is set" behavior ofgpu_instance_id.The sysfs interface facts this backend relies on (VF enablement via
sriov-manage, thecreatable_vgpu_typesformat,current_vgpu_typecreate/clear semantics, MIG ordering, attribute readability before/after a type is set) come from prior manual investigation on an H200 SXM system (vGPU Manager 580-series). This branch itself has not yet been run end to end on that hardware — I'd appreciate testing from anyone with Ada/Hopper vGPU hardware available before this is considered mergeable.make testandgolangci-lint runare clean.Known limitations (documented in the README)
Heterogeneous vGPU profiles (explicit
placement_idmanagement, mixed time-sliced profile sizes on one GPU) are not supported. The configuration format expresses type-name-to-count mappings only, matching the capabilities of the existing mdev backend.Framework detection is node-wide: one backend is selected for all GPUs on the node, so nodes mixing mdev-generation and vendor-VFIO-generation GPUs are not supported (same "one backend per node" model as before this change).
On GPUs whose vGPU devices are managed through the mdev framework but whose SR-IOV VFs are not enabled yet (e.g. Ampere before
sriov-manageruns), the system is indistinguishable from a vendor-VFIO one, so the vendor-VFIO backend may be selected. In that case the backend fails with an error pointing at the mdev framework; a subsequent run selects the mdev backend, since the mdev bus is populated once the VFs are enabled.Boot-time configuration is the reliable path. After live MIG re-configuration (without a reboot), the
creatable_vgpu_typesenumeration has been observed to come up empty on 580-series drivers; restarting the hostnvidia-vgpudservice may help. The Kubernetes DaemonSet applying the configuration at node startup fits the reliable path naturally.Unbinding a VF resets its
current_vgpu_typeto 0. MIG mode persists across reboots, but MIG instances, VFs and vGPU devices do not and are re-created on every boot by the manager.NVML must be loadable wherever
nvidia-vgpu-dmruns forassertto resolve already-applied configurations on vendor-VFIO systems. In Kubernetes this means using a deployment that mounts the driver installation and setsLD_PRELOAD, likeexamples/nvidia-vgpu-device-manager-mig-support-example.yaml.