[Performance] CUDA Conv: cuDNN frontend >= 1.24 calls cudaGetDeviceProperties on every plan rebuild (~1 ms per Conv node per input-shape change) #33769

Describe the issue

After moving from ORT 1.27.1 to 1.28.2, CUDA EP inference of a model with dynamic input shapes became 22-42% slower end to end. The extra time is all kernel (sys) time. Runs with a repeated input shape are as fast as before. Only runs whose input shape differs from the previous run are slower.

Root cause. On a shape change, CUDA Conv (and ConvTranspose) builds a new cudnn_frontend::graph::Graph for the node (conv.cc#L314-L316 → #L449 → #L129) and then calls check_support() on it.

• ORT 1.27.1 pinned cudnn-frontend v1.12.0. In that version, check_support() only checks the plans.
• ORT 1.28.2 pins v1.24.0. There, check_support() first calls context.populate_sm_version_from_device(). Each new graph starts with target_sm_version = -1, so this calls cudaGetDeviceProperties() every time. ORT's patched shim calls it directly, not through dlsym.
• Because ORT builds one graph per node, that is one cudaGetDeviceProperties() call per Conv node every time its input shape changes. Only the latest graph is kept, so alternating between two shapes rebuilds on every run.
• main still pins v1.27.0, which has the same code. So do the 1.29.0, 1.30.1 and 1.31.1 releases (v1.24.0 / v1.27.0 / v1.27.0).
• cudnn-frontend v1.32.0 switched to cudaDeviceGetAttribute() ([Auto-FE] shim: call ABI-changed CUDA runtime entry points with the loaded runtime's prototype (#1563) NVIDIA/cudnn-frontend#1566).

cudaGetDeviceProperties() is slow because it fills the whole struct. Here it took 0.86-0.99 ms per call, almost all of it sys time.

Measurements. All numbers are from the same host, RTX 5060, comparing 1.27.1 with 1.28.2:

The cost grows with the number of input-shape changes between runs, not with the length of the audio (1,543 = 20 runs × 77 nodes + 3 one-time calls). In this workload every VAD segment has a new length, so every segment costs 77 × ~0.95 ms ≈ 75 ms.

The CUDA EP options do not help. cudnn_conv_algo_search HEURISTIC/EXHAUSTIVE/DEFAULT, cudnn_conv1d_pad_to_nc1d, cudnn_conv_use_max_workspace=0, use_tf32=0, fuse_conv_bias=1 and arena_extend_strategy all leave the count at 1,543.

Confirmation. An LD_PRELOAD interposer returned a per-device cached cudaDeviceProp after the first real call. Results:

• the alternating run dropped from 2.70-2.75 s to 1.29-1.40 s, the same as 1.27.1 (1.30 s);
• end to end went back to 28.4 s and 61.3 s, with sys time back to 1.27.1 levels;
• the outputs were byte-identical.

The proposed fix is the same idea inside ORT: set the SM version on each graph from the device properties the EP already caches:

s_.cudnn_fe_graph = std::make_unique<cudnn_frontend::graph::Graph>();
const cudaDeviceProp& device_prop = GetDeviceProp();
s_.cudnn_fe_graph->set_sm_version(device_prop.major * 10 + device_prop.minor);

populate_sm_version_from_device() returns early when the version is already set. Graph::set_sm_version exists in every pinned version (e.g. v1.24.0). The value is the one cudnn-frontend would compute itself. In v1.24-v1.27, check_support() only uses it to choose the OSS SDPA / RmsNorm+SiLU engines, so Conv behaviour does not change.

Bumping cudnn-frontend to >= 1.32.0 would also fix this, but it is a larger change. Draft #33123 (LRU cache of Conv plans) helps when shapes recur. It does not help when every shape is new, as with variable-length speech segments, so the two changes complement each other. I have a PR ready for the set_sm_version change.

To reproduce

1. Take any model with Conv on a dynamic axis (here: a FastConformer/Parakeet ASR encoder, fp32, 77 Conv nodes). Run it with the CUDA EP, giving the input a different length on each call (or alternating between two lengths).
2. Count cudaGetDeviceProperties calls with this interposer (gcc -shared -fPIC -O2 count_gdp.c -o libcount_gdp.so -ldl, then run with LD_PRELOAD=./libcount_gdp.so):

#define _GNU_SOURCE
#include <dlfcn.h>
#include <stdio.h>
#include <time.h>
static long n, ns;
static int call(const char* name, void* prop, int dev) {
  int (*real)(void*, int) = (int (*)(void*, int))dlsym(RTLD_NEXT, name);
  struct timespec a, b; clock_gettime(CLOCK_MONOTONIC, &a);
  int r = real(prop, dev);
  clock_gettime(CLOCK_MONOTONIC, &b);
  __atomic_add_fetch(&n, 1, __ATOMIC_RELAXED);
  __atomic_add_fetch(&ns, (b.tv_sec - a.tv_sec) * 1000000000L + (b.tv_nsec - a.tv_nsec), __ATOMIC_RELAXED);
  return r;
}
int cudaGetDeviceProperties(void* p, int d) { return call("cudaGetDeviceProperties", p, d); }
int cudaGetDeviceProperties_v2(void* p, int d) { return call("cudaGetDeviceProperties_v2", p, d); }
__attribute__((destructor)) static void report(void) {
  fprintf(stderr, "cudaGetDeviceProperties: %ld calls, %.1f ms\n", n, ns / 1e6);
}

Expected: a few calls per process. Actual with 1.28.2: (number of Conv nodes) × (number of input-shape changes between runs).

The same pattern also shows up without ORT. A standalone program uses cudnn-frontend 1.24.0 with ORT's cudnn_frontend_win_dynamic_loading.patch applied, and builds one fp32 conv_fprop graph per shape the way CreateCudnnFeExecutionPlan does (validate, build_operation_graph, create_execution_plans, check_support, build_plans). For 100 graphs:

• it made 101 cudaGetDeviceProperties calls (~0.9 ms each);
• with set_sm_version(120) it made 1 call, and the time per graph dropped from 2.15-2.28 ms to 1.29-1.67 ms.

Urgency

Not blocking. It is a 22-42% throughput regression for dynamic-shape Conv models (speech) on the CUDA EP, starting in 1.28.

Platform

Linux

OS Version

Debian 13 (LXC container with GPU passthrough), kernel 6.17

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.28.2 (gpu_cuda12 prebuilt). 1.27.1 for comparison. main @ b5fe413 still pins cudnn-frontend v1.27.0.

ONNX Runtime API

C (through sherpa-onnx)

Architecture

X64

Execution Provider

CUDA (device_id=0, cudnn_conv_algo_search=HEURISTIC, other options default)

Execution Provider Library Version

CUDA 12.9 runtime, cuDNN 9.27, driver 580.126.09, GeForce RTX 5060 (sm_120)

Model File

NVIDIA Parakeet-TDT style FastConformer encoder exported to ONNX (fp32); any Conv model with a dynamic spatial/time axis shows the call pattern

Is this a quantized model?

No