Evaluating Logical Structure in Computer Programs Using LLMs

Abstract

Code comprehension theories postulate that programmers need to identify the logical steps of a computer program. This work examines the ability of large language models (LLMs) to identify and explain logical steps and their corresponding blocks of code in well-structured programming tasks. To evaluate the LLMs’ performance, we compare the LLM-identified logical steps with those identified by human experts. We assess the match between human and LLM annotations on this task using automated similarity analysis under multiple alignment strategies. Results show that the highest similarity between LLM-generated and human expert descriptions of the logical steps reaches up to 64.4%.