A Study on How Free Use of LLMs Assists Novices with Code Comprehension Tasks

Abstract

Code comprehension is a critical skill for computer science students who spend a substantial portion of their time engaged in reading and understanding code. While prior research has explored students’ use of Large Language Models (LLMs) for tasks such as code generation or bug fixing, there is very limited understanding of how effectively these students can prompt LLMs to get help for code comprehension activities. In this paper, we present a novel study exploring how intro-toprogramming students, i.e., novices to programming, freely prompt LLMs for code explanations. The goal was to understand how well LLMs can support students’ code comprehension activities with no training on advanced LLM prompting techniques. Our analysis reveals that while students’ prompts vary significantly, the quality of the LLM-generated code explanations for typical intro-to-programming code examples was considerably accurate, and complete. Students primarily use three types of prompts: whole-program explanation, specific logic explanation, and conceptual explanation while interacting with LLM. We also observed that access to LLM assistance is associated with a statistically significant increase in students’ confidence and improvements in code comprehension tasks.