Takuan: Root Causing Order-Dependent Flaky Tests through Analysis of Dynamic Invariants

Abstract

Automated testing is crucial to the development of stable, effective software. It allows developers to confidently make changes while preventing regressions in functionality. When tests can both pass and fail on the same version of code, they are known as “flaky tests.” Some flaky tests, known as order-dependent (OD) flaky tests, will pass or fail depending on test execution order. Prior work has investigated using dynamic tracing information to root cause flaky tests, but no work has utilized dynamic invariants, which are conditions that always hold true at a given point in a program (e.g., a field always has a certain value). We present Takuan, a framework for analyzing and root causing OD tests using dynamic invariants. Given an OD test, Takuan finds differences in the invariants collected from passing and failing runs, outputting “problem invariants”. Takuan then searches for methods that influence the problem invariants. We tested Takuan on 11 OD tests from 11 open-source projects taken from a public dataset. Takuan correctly found a problem invariant in five of the tests and indicated an influential method in three of the tests. Our work implies that dynamic invariants may be an effective way to root cause OD tests.