Large Language Model (LLM) Agents Achieve High Labor-Cost Reductions in Short-to-Medium Difficulty Software Engineering Tasks but Face Diminishing Returns on High-Difficulty Tasks

Abstract

As large language model (LLM) agents demonstrate an increasing role in software engineering tasks, their cost-effectiveness compared to human developers remains a key factor to be assessed. This study investigates the reduction of labor costs achieved by LLM-agent systems (GPT-4, Claude 3 Opus, and Gemini 3 Pro) when evaluated against the standards of human software developers controlling for task complexity and resolution rates. Task-completion data from SWE-Bench and METR's Human Calibrated Autonomy Software Tasks (HCAST) was collected and a cost-savings analysis with logistic regression was used to model the relationship between task difficulty and resolution likelihood, helping explain observed variation in savings across models and difficulty levels. Our results show that Gemini 3 Pro achieved substantially higher task resolution rates than GPT-4 and Claude 3 Opus across nearly all task difficulty categories (e.g., 88.14% on tasks under 15 minutes and 75.86% on 15-minute-to-1-hour tasks), producing substantial financial savings per task (e.g., averaging $128.65 saved for 1-4 hour tasks and $435.32 for tasks exceeding 4 hours). In contrast, GPT-4 and Claude 3 Opus resolved only a small fraction of tasks across all difficulty tiers (ranging from 0% to 13.40%), resulting in net financial losses per task in most categories, as generation costs were not offset by successful task resolution. However, performance and economic gain decreased significantly on more difficult tasks for all three models, and on models with lower success rates, such as GPT-4 and Claude 3 Opus (0% success rate on tasks greater than 4 hours). These findings demonstrate that while current LLM agents can serve as cost-effective substitutes for short-to-medium duration tasks when resolution rates are sufficiently high, task difficulty and the consistency of resolving these tasks remains a constraint on their overall positive economic impact and varies substantially across models.