fix(cuda): reach pinned-DMA bandwidth for large host-to-device uploads

[matteospanio]

Problem

Large uploads (create_from_slice / create) reach only a fraction of the device's transfer bandwidth. On an RTX 3070 (PCIe 3.0), a 268 MB f32 upload moves at ~0.9 GB/s, while the read path moves 6.5 GB/s on the same box — a 7× H2D/D2H asymmetry.

Cause: in write_to_gpu, contiguous transfers ≥ 100 MB from unpinned host memory go straight to memcpy_htod_async from pageable memory, which the CUDA driver stages internally and serially. Two smaller costs compound it: create_from_slices re-copies its (already owned) Vec into a fresh Vec before wrapping it into Bytes, and create_with_data page-locks a full-transfer-size staging buffer, bypassing the 100 MB size policy that reserve_cpu already encodes.

Fix

1. write_to_gpu: large contiguous transfers from unpinned memory are pipelined through two pooled 8 MB pinned staging chunks — each chunk's host memcpy overlaps the previous chunk's DMA. Falls back to the direct path when the pinned pool cannot serve the chunks. Behavior for < 100 MB and already-pinned buffers is unchanged.
2. create_from_slices: drop the redundant full-size host copy (the slice data is already an owned Vec at that point).
3. create_with_data: route staging through reserve_cpu so the existing size policy applies.

Measurements (RTX 3070, PCIe 3.0, 268 MB f32)

	before	after
create(Bytes::from_shared(..)) + sync	298 ms (0.9 GB/s)	42.3 ms (6.35 GB/s)
read (unchanged, for reference)	41 ms (6.5 GB/s)	41 ms (6.5 GB/s)

Measured with the same patch backported onto v0.10.0 (identical code in this area). Byte-exact roundtrips verified at 4 KB, 8 MiB, 8 MiB+1, 16 MiB−3, 100 MiB+4 KiB+1, and 268 MB via both create_from_slice and create(Bytes::from_shared(..)); also validated by a downstream GPU DSP test suite (bit-parity of kernel results vs CPU reference).

🤖 Generated with Claude Code

[nathanielsimard]

I'm not sure the current fix is the right one. We already haven a staging utility exposed from the client, I think we should use that instead, since that will improve performance across all backends.

[shurkus]

A data point from a downstream user, on the suggestion to use the staging utility instead.

We tried exactly that from the caller side. ComputeClient::staging() does engage — and
both creation paths in client.rs then discard the result one line before the DMA:

// do_create
self.staging(data.iter_mut(), true);
...
Bytes::from_bytes_vec(data.to_vec())   // pinned -> pageable, right before server.write

// do_create_from_slices
Bytes::from_bytes_vec(data.to_vec())   // same, and staging() is never called here

Measured on a V100-SXM2 over PCIe 3.0 x16, 3.07 MB transfers, 200 reps, idle machine
(VRAM control 770.7 GB/s, on record for this card):

• client.create — 1.96 GB/s
• staging() + create — 1.58 GB/s; the receipt says handed over Pinned, so the arm
  really did engage, it is just slower for the extra copy
• the same transfers through the bare CUDA driver API, pinned + reused device buffer —
  12.05 GB/s

So the staging utility is already public and already reachable; what a caller cannot do
today is keep the pinning alive as far as the DMA. Whatever shape the fix takes, that copy
is what stands between the two numbers above.