RAMP

Exploring the Feasibility of Detecting Physics Student Misconceptions in Writing Assignments Using Large Language Models

Abstract

Students in introductory STEM courses frequently have misconceptions about the material. Writing assignments can help instructors identify these, but are often impractical and time-consuming to grade, especially in large classes. In this study, we curated student responses from an introductory physics assignment based on a misconception related to motion. We formulated the task of identifying misconceptions within a sentence as a binary classification task and developed the RAMP (Reporter of Aggregated Misconceptions in Physics) classifier using ModernBERT. Experimental results indicate that RAMP is effective for identifying student misconceptions, noticeably outperforming various prompting techniques using several LLMs and traditional machine learning classifiers. With the refinement of hyperparameters and additional data, RAMP may be improved to an acceptable level, where it can be used as the back-end of an instructor-facing tool that reports student misconceptions across writing assignments in introductory physics courses.