We should investigate how to track quality metrics for perf comparison:
- number of ambiguous comparison results
- number of false positives
After making changes to warmup run number, stopping criterion, and comparison scripts, we should be able to evaluate the change by tracking these metrics.
This issue can be close with an outline of how this tracking can be implemented.
We should investigate how to track quality metrics for perf comparison:
After making changes to warmup run number, stopping criterion, and comparison scripts, we should be able to evaluate the change by tracking these metrics.
This issue can be close with an outline of how this tracking can be implemented.