A Preliminary Study of Test Flakiness in Large Language Model Generated Tests

Abstract

Developers trust automated tests to catch bugs before they reach users, yet a growing class of tests undermines that trust entirely: flaky tests, which pass and fail unpredictably on the same version of code. The rise of test generation using large language models (LLMs) raises a critical question the field has not yet answered — do these tools generate tests that are flaky? To find out, we run popular large language model based test generation tools on four open-source Java projects. Across 182 generated tests with a 93% pass rate, one confirmed flaky test was found. This flaky test used a HashMap and assumed a fixed data ordering that Java does not guarantee. To better understand the flakiness of LLM-generated tests, future work should evaluate test generation on focal methods that are known to have resulted in flaky tests. This line of future work can shed more light into the flakiness of LLM-generated tests and provide insights into what characteristics commonly occur in focal methods that result in flaky tests. Our preliminary findings show that LLM-generated tests carry a distinct flakiness risk rooted in the model's incorrect assumptions about Java runtime behavior, which is a risk that grows more consequential as LLM testing tools see wider adoption.