Skip to content

CI: run the CUDA backend tests on a simulated GPU (optional workflow) - #488

Open
saqibkh wants to merge 2 commits into
stotko:masterfrom
saqibkh:ci-simulated-gpu
Open

saqibkh wants to merge 2 commits into
stotko:masterfrom
saqibkh:ci-simulated-gpu

Conversation

@saqibkh

@saqibkh saqibkh commented Oct 8, 2026 •

Copy link
Copy Markdown

Disclosure: I am the author of PantheonSim (https://github.com/pantheongpu/pantheonsim) and of the setup-pantheonsim action used here. Please weigh this accordingly.

What this does
CI currently builds and tests only the OpenMP backend ("Tests OpenMP"); the CUDA backend is never executed because hosted runners have no GPU. This adds a separate workflow, tests_cuda_simulator.yml, that builds the CUDA backend and runs the existing test suite (tools/backend/configure_cuda.sh, tools/build.sh, tools/run_tests.sh, unchanged) on a simulated NVIDIA A100 on ubuntu-24.04.

  • Triggers: workflow_dispatch, weekly schedule, and PRs touching the CUDA backend, impl, tests, CMake files or the workflow itself. It is a separate workflow and not part of any required check; I'd suggest keeping it that way.
  • permissions: contents: read, persist-credentials: false, 60 min timeout.
  • Every third-party action is SHA-pinned (actions/checkout v7.0.1, pantheongpu/setup-pantheonsim v0.1.3); your existing dependabot config for actions will propose bumps.
  • It builds with -DSTDGPU_BUILD_SHARED_LIBS=ON (the simulator provides a shared cudart; with static stdgpu libraries the tests fail) and -DCMAKE_CUDA_ARCHITECTURES=80-virtual (PTX only, which the simulator JIT-compiles).

Result: the CUDA test suite (702 tests) passes on the simulated A100 in about 5 minutes of test time, run exactly as the workflow does (configure_cuda.sh with the flags above, build.sh, run_tests.sh) against the simulator commit the pinned action builds. Local caveat: I used Ubuntu's CUDA 12.0 toolkit with g++-12 rather than the 12.6 the workflow installs, and I could not run the workflow on GitHub itself without pushing.

Tradeoffs, honestly

  • A simulator is functional, not bit- or timing-accurate: it cannot catch hardware- or driver-specific bugs. A red run may be a stdgpu bug or a simulator bug, which adds triage work. It complements real-GPU testing and does not replace it.
  • The action downloads and builds a third-party project (the simulator, at a commit pinned by the action release) and installs the CUDA toolkit from NVIDIA's apt repo. The SHA pin means nothing changes until bumped, but you are trusting an additional small project that I maintain. If it becomes a burden, deleting the one workflow file removes it entirely.
  • configure_cuda.sh enables warnings-as-errors; a different toolkit than the one I tested with could surface new warnings.

Happy to change triggers, GPU model, toolkit version, or drop the schedule.

Update: re-pinned to setup-pantheonsim v0.1.3 and re-ran this job locally against that release's simulator: same results as above. That release's simulator no longer ships NVIDIA's cuDNN headers in its source tree; they are downloaded from the nvidia-cudnn-cu12 wheel when it is built, so the first build (or any cache miss) fetches a wheel of several hundred MB.

Adds a separate, non-required workflow (manual, weekly and on CUDA-related changes) that builds the CUDA backend with shared libraries and runs the test suite on a simulated NVIDIA A100 via pantheongpu/setup-pantheonsim, pinned by commit SHA. Uses read-only permissions.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant