An Experience Report of Applying Automated Techniques to Non-deterministic Bugs in Software Testing

Abstract

Software testing is widely used by developers to reduce bugs in software. Flaky tests are tests that non-deterministically pass and fail when run on the same version of code, and they greatly impact the software industry in terms of time and monetary losses. Additionally, failures due to flaky tests mislead developers about recent changes and reduce developers' trust in software testing. Tests can be flaky due to concurrency, network calls, asynchronous calls, dependency on the test order, and many other reasons. Order-Dependent (OD) flaky tests are a specific subset of flaky tests that pass and fail only due to the order in which the tests are run.

Our work focuses on fixing OD Python flaky tests found in popular open-source projects using iPFlakies, a framework that can automatically detect OD tests and generate patches for the OD tests. Our work consists of checking to see if the detected tests are OD at the latest version of the projects, and if so, applying and improving (e.g., removing lines of code that do not directly contribute to the fix) the automatically-generated patch. We prepared pull requests for 10 tests across 8 open-source projects found on GitHub, and developers from two different projects have already accepted our pull requests. Our work will help prevent flaky tests from negatively impacting the developers of these projects in the future.