Add 100 minute timeout to C++ ctest runs

[bdice]

Wraps the sharded C++ ctest run in timeout, sending SIGINT then SIGKILL after 60s, to match the Python test timeouts from #2734.

The slowest sharded C++ job across the last 20 passing PR runs took 85 minutes (V100 shard 3), so 100 minutes leaves headroom without letting a hung job run to the 6h limit.

[bdice]

Data behind the 100 minute value

Whole-job wall time (includes environment setup) of the sharded conda-cpp-tests jobs in the 20 most recent successful pr.yaml runs (288 jobs):

• Overall: median 27m, max 85m.
• Slowest shard is shard 3. Max by GPU: V100 85m, H100 65m, A100 61m, RTX PRO 6000 60m, L4 47m.

V100 only (12 jobs per shard):

Shard	Min	Median	p90	Max
1	26m	35m	65m	78m
2	10m	13m	23m	34m
3	31m	47m	74m	85m
4	29m	36m	44m	51m

100m leaves at least 15m of headroom over the slowest job seen. These durations include setup, so the ctest step itself is shorter.

[bdice]

Jobs that hit the 6 hour limit in the last 30 days

Scope: failed or cancelled pr.yaml and test.yaml runs created since 2026-09-05. GitHub reports a job timeout as cancelled, so I counted cancelled jobs that ran 355 minutes or more. Runs that succeeded cannot contain a hung job.

28 test jobs hit the limit, across 19 distinct runs (22 in pr.yaml, 6 in test.yaml):

Job	Count
conda-cpp-tests	9
conda-python-tests	13
wheel-tests-cuvs	6

The 9 C++ jobs are in four runs: 37055712451 (1 job), 37345134834 (2), 37353917582 (4) and 37358545509 (2). They span V100, A100, H100 and L4.

With a 100m limit each would have ended about 260 minutes sooner, roughly 39 GPU-hours across these nine jobs.

Not covered: in-progress runs, and the 10 job logs that could not be downloaded.

[bdice]

NEIGHBORS_DYNAMIC_BATCHING_TEST is the test that hangs

Source: logs of the failed or cancelled conda-cpp-tests jobs from the last 30 days that ran at least 100 minutes (182 jobs; 39 of the 49 without a ctest summary were readable, the other 10 log downloads failed).

Normal runtime: in the 133 jobs that reached a ctest summary, the test always passed, in 4.1m median (p90 5.3m, max 6.3m).

Hang: in 10 jobs from the sharded layout, this test was the only one that had started but not finished (8 started, 7 finished); nothing else was running. All 10 are shard 4 (test #28 of that shard) on V100, A100, H100 and L4:

Run	Jobs	Job time	Time in the test
37055712451	V100	360m (6h limit)	~334m
37353917582	V100, A100, H100, L4	360-365m (6h limit)	~335-341m (V100: 158m measured, which looks off)
37358545509	H100, L4	360m (6h limit)	~338-341m
37358545509	A100	350m	18m (different pattern, not counted as a 6h hit)
37378638396	H100, L4	112-116m	~92-94m (cancelled while still hung)

So 7 of the 9 C++ jobs that hit the 6h limit were stuck in this test. The other 2 (V100 and A100 in run 37345134834) never reached the tests: both stalled in setup, as did another job from run 37378638396 (H100, 112m) cancelled while downloading the artifact.

Older runs: before sharding, 8 jobs were cancelled with this test as the only unfinished one, after 137-213 minutes in the job. In each, the cancel came 0-4 minutes after the test started, in line with its normal 4 minute runtime, so I do not treat those as hangs. No job in the older runs (before 2026-10-02) reached the 6h limit in the C++ tests.

Since the hung jobs are all shard 4, a ctest --timeout would name the test and let the rest of the shard finish, but the 100m wrapper in this PR already bounds the hang. I have not added a per-test timeout here. The stalls in setup are not covered by either.

[bdice]

/merge

[coderabbitai]

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

⚙️ Run configuration

• Configuration used: Repository: NVIDIA/cuvs/.coderabbit.yaml
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 2ebdf8bb-e287-4c62-9864-4084530effe7

📥 Commits

Reviewing files that changed from the base of the PR and between 74ed86d and 21e0c75.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: NVIDIA/cuvs/.coderabbit.yaml
• Review profile: CHILL
• Plan: Enterprise
• Run ID: a8bb56e0-7449-4f67-959c-4b043815d160

📥 Commits

Reviewing files that changed from the base of the PR and between 6cdea7e and 74ed86d.

📒 Files selected for processing (1)

• ci/test_cpp.sh

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

---

📝 Summary

Summary by CodeRabbit

• Chores
  • CI test runs now have a 100-minute time limit. If a run reaches the limit, it receives up to 60 seconds to exit before being forcefully stopped. This helps ensure stalled test jobs finish predictably rather than blocking the validation process indefinitely.

Walkthrough

The libcuvs ctest invocation now runs under timeout with a 100-minute limit, SIGINT at the limit, and SIGKILL after a 60-second grace period if the process does not exit.

Changes

CTest timeout handling

Layer / File(s)	Summary
Bound the CTest run ci/test_cpp.sh	The ctest invocation now uses timeout with a 100-minute limit, sends SIGINT at the limit, and sends SIGKILL after 60 seconds if the process remains running.

Priority: ➖ Normal

Estimated code review effort: 1 (Trivial) | ~5 minutes

Change: Other

Merge Risk: ⚪ Minimal · up to 74ed8

The timeout is intended to bound hung C++ tests, and a timeout failure is reported as a CI failure. No merge-blocking risk was identified.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely identifies the main change: adding a 100-minute timeout to C++ ctest runs.
Description check	✅ Passed	The description explains the timeout behavior and its rationale, including the reported job timings and the goal of limiting hung runs.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

🧪 Generate unit tests (beta)

• Create a new PR

🛠️ Fix failing CI checks 💡

• Commit to this branch
• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}