Enhancing Flaky Test Detection Using iDFlakies and TuscanE

Abstract

Flaky tests are software tests that lead to inconsistent results, i.e., tests that pass and fail on the same version code. Prior work found that order dependent tests (OD) are one of the most prominent categories of flaky tests. OD tests are tests whose outcomes depend on the order they are run, while non-order dependent tests (NOD) are tests whose outcomes do not depend on the order they are run. Two tools from prior work to detect flaky tests are iDFlakies and TuscanE. iDFlakies is a tool that detects flaky tests by rerunning tests in various orders, detecting whether they are flaky, and categorizes tests as OD or NOD. Previous work has shown that iDFlakies can detect some flaky tests by running tests in randomly generated test orders. TuscanE is a tool that systematically generates test orders, where once all generated orders are run, will also guarantee the detection of certain OD tests. To help improve the detection of OD tests in iDFlakies, we integrate the two tools together to enable one to detect OD tests more efficiently. Our evaluation of the combined tool that systematically generates test orders is able to detect OD tests faster than using iDFlakies with random test orders.