Generating Efficient Test Suite Orders to Improve Test Suite Runtime

Abstract

Regression testing is challenging to scale in large, complex systems due to the high cost of running tests as developers make frequent changes to their codebase. To address this problem, prior research proposed techniques to reduce the test suite runtime by running fewer tests, either by selecting to run only a subset of tests affected by changes, or by removing redundant tests w.r.t. some criteria. However, such techniques may miss the detection of faults in future changes if the key test that detects the fault is not run. Recent work has found that running tests in different test-orders can have a large impact on test runtime. In this work, we propose new approaches to generate test-orders that can reduce test runtimes on future versions of code. By running tests in an efficient test-order, yet still running all tests, we can reduce the overall runtime of tests without compromising their fault-detection capability. We explore different approaches to sort tests based on metrics traced from live runs. These metrics include test runtime, total heap allocation, and time spent on JIT compilation. These approaches work by running tests in a few different test-orders, under additional instrumentation, then using the collected information to generate more efficient test-orders to use. We evaluate the mean values of metrics from five different runs of ten test suites from popular open source projects. Compared to running arbitrary test-orders, our results showed a statistically significant speedup for four out of ten test suites. Of those four of ten test suites, the mean speedup was 17.6%. The highest speedup was 24.1%, coming from a strategy that sorts based on heap allocation. Given our promising preliminary results, codebase maintainers may find it useful to collect certain metrics and generate efficient test orders during development.