Detecting and Understanding Operating Systems Dependent Flaky Tests

Abstract

Developers typically run tests after every code change. Operating System (OS) dependent flaky tests are tests that can non-deterministically pass and fail on the same version of code, depending on the operating system in which they are executed in. These flaky tests can mislead developers about their recent code changes and waste their time. To detect and understand these flaky tests in open source projects, developers and researchers have to compile and run tests on various operating systems in the cloud through tools, such as Github Actions, which can be costly in time and money. To help reduce the cost of detecting OS dependent flaky tests, stakeholders can compile and run potential OS dependent flaky tests locally as opposed to relying on cloud machines. By utilizing a new tool which compiles and runs tests locally, the process of repeatedly running potential OS flaky tests in different environments (e.g., different operating systems) can require drastically less time and resources. To achieve this reduction, we use the nektos/act tool, which simulates Github Actions locally, on different open-source projects, such as Apache Dubbo and Dropwizard. Limitations with the tool, such as its inability to consecutively run tests on different operating systems and its inconsistent output in some test scenarios, were discovered. These insights can help guide revisions to the tool so that it better meets users’ needs.