Personalized Debugging Feedback: A Comparative Study of AI Assistants for Novice and Expert Programming Tasks on CodeForces

Abstract

Code debugging is an essential skill for a student programmer that must be applied efficiently and effectively. However, teaching debugging techniques in a classroom setting is sporadic [1]. Although Artificial Intelligence (AI) has been widely integrated into various computer science tasks, no analysis was found on its effectiveness in debugging code for varying levels of programming expertise. After conducting a literature review, ChatGPT, Github Copilot, Phind, and Amazon Q were selected as notable AI debuggers. This study evaluated the efficiency and effectiveness of these 4 AI models on debugging code submissions for 18 CodeForces tasks, ranging from novice (800 rating) to advanced (1500 rating). The code simulated by each AI model was evaluated by four student researchers on a 500-point rubric on their accuracy (200 points), comprehensibility (100 points), code explanation quality (100 points), and similarity of code logic to the user’s inputted solution (100 points), and the average was used for analysis. The results indicate that GitHub CoPilot and ChatGPT displayed the highest overall effectiveness for novice debugging tasks, with an average score of 455.55 out of 500 points (91.1%), and Amazon Q exceeded other AI models for advanced debugging tasks, with an average score of 444.44 out of 500 points (88.9%). Overall, ChatGPT had the highest average score across both novice and advanced programming questions, with an average score of 438.89 out of 500 points (88.0%) on the 18 coding tasks. These findings demonstrate the strong potential for AI models to assist with debugging, specifically, for novice programming tasks. Future research would involve expanding the dataset to include various programming languages and levels of expertise, enabling the development of enhanced systems for bug resolution. 

[1] Noller, Y., Chandra, E., HC, S., Choo, K., Jegourel, C., Kurniawan, O., & Poskitt, C. M. (2025), Simulated Interactive Debugging, arXiv, https://arxiv.org/abs/2501.09694