Problem
This is a subissue of #408.
NVBench currently uses the stdrel stopping-criterion parameter --min-time for two separate purposes:
- In cold and CPU-only measurements,
--min-time controls when the stdrel stopping criterion is allowed to stop based on noise convergence.
- In hot/batched measurements, the same value is also used to estimate batch size and decide how much accumulated GPU time the batched measurement loop should collect.
These are different concerns and should not be coupled.
For cold measurements, --min-time is part of a convergence policy: collect enough samples before trusting relative standard deviation.
For hot/batched measurements, the target duration is a batching policy: choose enough repeated kernel launches to measure warm-cache / steady-state behavior with reasonable timing resolution.
Those two values may reasonably need to be different.
Why this matters
1. Cold convergence time and hot batch target time may not want the same value
A user may want cold measurements to collect longer for stable noise estimates, while keeping batched measurements short:
Today, that also increases the batched measurement target duration.
Conversely, a user may want a larger hot/batched target duration without forcing stdrel cold measurements to wait longer before convergence is allowed.
There is currently no separate control for that.
2. Batch-size control is lost when using a non-stdrel stopping criterion
Because --min-time belongs to the stdrel stopping criterion, it is not available when another stopping criterion is active:
./bench --stopping-criterion entropy --min-time 0.1
./bench --stopping-criterion sample-count --target-samples 20 --min-time 0.1
Both of the above fail because entropy and sample-count criteria do not accept --min-time parameter.
But the user may still need to control the hot/batched measurement loop target duration. That should not depend on the selected stopping criterion which applies to cold measurements loop.
Desired Behavior
NVBench should provide a dedicated way to control the hot/batched measurement loop target duration independently from stdrel. That control should:
- apply to batched measurements directly
- be valid regardless of the selected stopping criterion
- not change cold or CPU-only stopping behavior
- compose cleanly with
--min-samples and --timeout
- support global and per-benchmark option scoping
Proposed Direction
Introduce a dedicated benchmark option for the hot/batched measurement loop, for example:
--batch-target-time <seconds>
This option would control the target accumulated GPU time for batched measurements, while --min-time would remain a stdrel stopping-criterion parameter.
Also see: #408 (comment)
Problem
This is a subissue of #408.
NVBench currently uses the
stdrelstopping-criterion parameter--min-timefor two separate purposes:--min-timecontrols when thestdrelstopping criterion is allowed to stop based on noise convergence.These are different concerns and should not be coupled.
For cold measurements,
--min-timeis part of a convergence policy: collect enough samples before trusting relative standard deviation.For hot/batched measurements, the target duration is a batching policy: choose enough repeated kernel launches to measure warm-cache / steady-state behavior with reasonable timing resolution.
Those two values may reasonably need to be different.
Why this matters
1. Cold convergence time and hot batch target time may not want the same value
A user may want cold measurements to collect longer for stable noise estimates, while keeping batched measurements short:
Today, that also increases the batched measurement target duration.
Conversely, a user may want a larger hot/batched target duration without forcing
stdrelcold measurements to wait longer before convergence is allowed.There is currently no separate control for that.
2. Batch-size control is lost when using a non-stdrel stopping criterion
Because
--min-timebelongs to thestdrelstopping criterion, it is not available when another stopping criterion is active:Both of the above fail because
entropyandsample-countcriteria do not accept--min-timeparameter.But the user may still need to control the hot/batched measurement loop target duration. That should not depend on the selected stopping criterion which applies to cold measurements loop.
Desired Behavior
NVBench should provide a dedicated way to control the hot/batched measurement loop target duration independently from
stdrel. That control should:--min-samplesand--timeoutProposed Direction
Introduce a dedicated benchmark option for the hot/batched measurement loop, for example:
This option would control the target accumulated GPU time for batched measurements, while
--min-timewould remain astdrelstopping-criterion parameter.Also see: #408 (comment)