Predictive Identification of At-Risk Students: Using Learning Management System Data

Background

When discussing student success in higher education, the term “at-risk” is often used to describe students who may be struggling and are at risk of negative outcomes such as failing, dropping out, stopping out, or burning out (Chibaya et al., 2022). However, at-risk can also refer to students who are not meeting satisfactory academic progress (SAP) requirements. SAP is a standard of academic performance that students must maintain to be eligible for federal financial aid. This standard includes a minimum grade point average (GPA) and completing a certain percentage of courses attempted. SAP was first introduced in 1976 as an amendment to the Higher Education Act of 1965 (Federal Student Aid, 2022).

When building predictive models for students, researchers often use the term at-risk to divide the student population into two groups: those who are at risk and those who are not. In these models, several factors emerge as significant predictors of a student being at risk, including grades (Larose & Tarabulsy, 2014; Russell et al., 2020), preparedness (Casanova et al., 2021; Owen et al., 2021; Russell et al., 2020), student behavior/attitudes (Delmas & Childs, 2021; McManus, 2020; Owen et al., 2021; Russell et al., 2020), academic momentum (Adelman, 1999, 2006), and student academic and social integration (Tinto, 2012).

Accurate predictions of academic performance for individual students are essential to inform interventions. To achieve this goal, researchers have identified several important attributes for predicting student academic performance, such as aspects of a student’s demographic and socioeconomic background (e.g., place of birth, disability, parent academic and job background, residing region, gender, socioeconomic index, health insurance, frequency of going out with friends, financial status; Imran et al., 2019; Jain & Solanki, 2019; Purwoningsih et al., 2019; Rubiano & Garcia, 2016; Shanmugarajeshwari & Lawrance, 2016; Tenpipat & Akkarajitsakul, 2020; Zeineddine et al., 2021), pre-enrollment (e.g., high school or level 12 performance and grades, entrance qualification, SAT scores, English and math grades, awards, school they attended; Iatrellis et al., 2021; Imran et al., 2019; Jain & Solanki, 2019; Rubiano & Garcia, 2016; Tenpipat & Akkarajitsakul, 2020; Xu et al., 2017; Zeineddine et al., 2021), enrollment (e.g., enrollment date, enrollment test marks, number of courses students previously enrolled in, type of study program, study mode; Berens et al., 2019; Kemper et al., 2020), tertiary academic (e.g., attendance, number of assessment submissions, student engagement ratio, major, time left to complete degree, course credits, semester work marks, placements, count and date of attempted exams; Berens et al., 2019; Iatrellis et al., 2021; Imran et al., 2019; Kemper et al., 2020; Xu et al., 2017; Yang & Li, 2018), and LMS-based data.

While student records, such as GPA and semester or final results, have been frequently used as categorical variables to assess a student’s academic potential (Akram et al., 2019; Berens et al., 2019; Iatrellis et al., 2021; Imran et al., 2019; Jain & Solanki, 2019; Kemper et al., 2020; Li et al., 2020; Purwoningsih et al., 2019; Rubiano & Garcia, 2016; Tenpipat & Akkarajitsakul, 2020; Xu et al., 2017; Yang & Li, 2018; Zeineddine et al., 2021), few studies have used LMS-generated data to predict student achievement. LMS data, such as the frequency of interaction of a student with each module on LMS (Nespereira et al., 2015), counts of hits, forum post details, counts of assessments viewed and submitted on LMS (Purwoningsih et al., 2019), start and end dates, and assessment submission dates (Akram et al., 2019), provide complete information about a student’s engagement in online learning sessions and workshops. These data are automatically generated and stored by the LMS, making them cost-effective, accessible, and relatively easy to analyze.

However, few studies have investigated the correlations between LMS attributes, selection of relevant attributes, and tuning of classifier algorithm parameters for accurate prediction of student progress. Furthermore, most studies have focused on traditional face-to-face or online classroom settings (Li et al., 2020), and data sets obtained from blended learning are limited (Nespereira et al., 2015; Purwoningsih et al., 2019). To address these gaps, researchers must focus on early detection of at-risk students to implement remedial measures promptly.

Over the past decade, there has been a significant increase in the development of correlation and predictive analytics models that rely on data mining of student data to identify at-risk students. These models have been developed by researchers in various fields, including education, computer science, and data analytics, and have shown promising results in identifying students who may be struggling academically.

Some of the most notable studies in this area include those conducted by Akçapınar et al. (2019), Baron et al. (2022), Cardona et al. (2020), Essa & Ayad (2012), Gašević et al. (2016), Marbouti et al. (2016), Papamitsiou & Economides (2014), Trivedi (2022), and Wolff et al. (2013). These studies have demonstrated that by analyzing various student data points, such as grades, attendance records, and participation in extracurricular activities, it is possible to identify students who may be at risk of academic failure.

One of the key benefits of these models is that they enable educators to identify at-risk students at an early stage. This is crucial because research has shown that the earlier a student is identified as being at risk, the more likely it is that intervention strategies will be effective. For instance, several studies (Danilowicz-Gösele et al., 2017; Dweck et al., 2014; Gašević et al., 2016; Jayaprakash et al., 2014; Singell & Waddell, 2010) have shown that early intervention can help prevent academic failure and improve student outcomes.

Machine learning is the intersection between the fields of computer science and statistics (Jordan & Mitchell, 2015). Statistical methods can generate probabilities and provide statistically significant inferencing on data sets. However, as the size of the data being studied increases, the human capital required to perform such calculations quickly becomes out of reach. Computers provide the ability to rapidly and repeatedly execute calculations, allowing statistical analyses that could not humanly be performed to be calculated in minutes. A variety of machine learning models are available for use in performing analyses, and several more common and reliable models were chosen for comparison in this study, namely logistic regression, k-nearest neighbor (kNN), random forest, and a neural network (Cardona et al., 2020).

Logistic regression is a widely used technique in producing predictions for classification problems. It involves using the linear regression technique with a sigmoid function to convert the probabilistic outcome to a binary classifier, such as predicting whether a student will pass or fail a course.

Another popular classification model is kNN, which calculates the distance, usually Euclidian, of each new observation as it is added and identifies its “k” nearest neighbors. The classification of the majority of the nearest neighbors is then assigned to the new input. Taunk et al. (2019) explain that kNN models are particularly useful when there are many features but only a few are relevant to the classification task.

Random forest models, on the other hand, generate classifications through a majority voting process of a large number of decision trees. The individual decision trees are constructed using a random selection of the independent variables, as well as a sampling with replacement of observations. This approach provides a powerful way to handle high-dimensional data and is capable of capturing complex interactions between variables. Breiman (2001) suggests that the strength of random forest models is their ability to handle large data sets with high dimensionality.

Finally, a single-layer, feed-forward neural network is a model inspired by the human brain. It uses a set of interconnected nodes that resemble the neurons in the human brain to generate predicted values. The input layer of the model comprises the independent variables being used in the problem, which are weighted and transmitted to nodes in the hidden layer. Each node in the hidden layer receives input from each input node, and values are summed and processed by an activation function, often a sigmoid function like that used in logistic regression. In this single-layer neural network, outputs from the hidden layer nodes are collected by the output layer, which classifies the observation. Abiodun et al. (2018) note that neural network models are particularly useful when the relationships between the input variables and the output variable are complex and nonlinear.

Methods

To achieve the goal of identifying students likely to fail a course, this study was framed as a binary classification problem. Therefore, the output of the model needed to label a student as either at risk of failing or not at risk of failing based on available grade information after the fifth week of classes. The specific features of interest were the student’s weekly grade, representing their current standing in the course, and if the student had a history of missing grades/assignments.

The general approach taken in this research effort is reflective of the Open Academic Analytics Initiative (OAAI) predictive modeling architecture outlined by Jayaprakash et al. (2014). This involved performing traditional extract-transform-load (ETL) processing from the LMS to gather and prepare the data prior to inputting it into the machine learning algorithms. The ETL process retrieves data from the LMS and performs a series of programmatic actions to clean the data, manage missing values, convert data types, and derive features. Certain features were transformed post-extraction but prior to model training to provide greater flexibility in the analysis and modeling processes.

Grade information for the Spring 2022 semester was extracted from the university’s LMS, D2L Brightspace, and the Banner Enterprise Resource Planning (ERP) database, used by the Registrar to hold student records. The raw data extracted from the systems were composed of weekly data pulls reflecting students’ weekly standing in a course, as well as data containing students’ final semester grades. Files were provided in the form of Excel spreadsheets and comma-separated values.

Data transformation was performed using Python applications and a MySQL database. Weekly grade data were cleaned, parsed, transformed, and then stored in database tables along with final course grades present in the Banner ERP extract. Following this initial load, the data were analyzed to see if weekly grades were missing for certain assignments for each student in a course—once grades began to appear in the course. If a weekly grade was missing, a determination needed to be made as to a potential cause. For example, the absence of a grade in the LMS could be due to a student’s failure to submit an assignment in a timely manner, because there were no assignments due, or because the instructor had not graded any assignments that week. The data were scanned to determine the existence of grades for each course for each week. If grades were present, a separate database record was created to signify the existence of grades that week as well as the percentage of students who had received a grade. If a specified percentage of students had a weekly grade present in the course that week, students without grades were flagged as having missed work that week. This value is referred to as the missed grade threshold (MGT). If no grades had been entered or the percentage of students with a grade was less than the specified percentage, the student’s grade from the previous week was carried forward and used as the current week’s grade. Initially, an MGT value of 80% was selected for model comparison, and later values of 67, 75, 80, and 90% were used to check for differences in model performance.

With this level of cleaning and transformation complete, an extract from the MySQL database was taken to be used in modeling. The extract file contained records representing each student in each course, with columns containing weekly grades, flags indicating missing assignments for a given week, and the student’s final course grades, if present in the extracts. The data contained in this extract file were further analyzed and processed using R scripts. Given that the final course grade was used as the dependent variable, only observations with final grades consisting of a traditional A-F grade value, passing (P)/not passing (NP), incomplete (I), and withdrawal (W) were retained, and records with other final semester grade values were dropped from the data set. Students whose final grade was either a D, F, NP, I, or W were labeled with a “0” to denote an at-risk student, and those with grades of A, B, C, or P were labeled with a “1.” Since early detection of at-risk students was the goal, independent variables were the current course grade after the fifth week of the semester and the presence of missing grades in weeks 3, 4, and 5. We found that the data were too sparse before week 5 to have significant, meaningful information (see Figure 1).

Several supervised machine learning model approaches were utilized to construct models, and the resulting output of each was reviewed for the best performance. Approaches included logistic regression, random forest, kNN, and a single-layer neural network. RStudio was utilized as an Integrated Development Environment (IDE), and code was constructed primarily using TidyVerse (Wickham et al., 2019) and TidyModels (Kuhn & Wickham, 2020). The extract file was separated into training and testing data sets using a 70/30 split and the final course grade as a stratified sampling variable.

After training each of the models mentioned, predictions were created with the model using the testing data set. Students predicted to be at risk were labeled with a “0,” and those predicted as not at risk were labeled with a “1.” Confusion matrices and receiver operating characteristic curves were created, as were a metric set for each model to evaluate the accuracy, sensitivity, specificity, and precision. Furthermore, 10-fold cross-validation was performed using the same metrics to help ensure that the models were generalizable.

Each model was executed against the same training and testing data sets for each fold, and model performance statistics were gathered. The models were tuned based on model performance metrics, and additional runs were made before a final run was performed. In each case, the tuned models performed better than the untuned version. Finally, the tuned models trained using LMS grade book data from Spring 2022 were compared by additionally testing their performance on LMS grade book data from Fall 2021.

Limitations

One limitation of the study is that the data analyzed was collected from a single semester and was not randomized. This means that the findings may not be generalizable to other semesters or institutions. For example, the study may not have captured the impact of seasonal factors on student performance, such as the effects of weather or holidays on attendance and motivation. Additionally, the lack of randomization means that the sample may not be representative of the broader student population at the university, which may limit the generalizability of the findings.

Another limitation of the study is the lack of prespecified subgroups, which may limit the ability to draw specific conclusions about certain student populations. For example, the study did not investigate whether there were differences in the factors that contribute to success or failure between students from different majors or with different academic backgrounds. This limitation may limit the applicability of the findings to specific student groups and may make it difficult to design targeted interventions that are tailored to the needs of different student populations.

Another limitation is the variability in the use of LMS grade books across courses and disciplines, which may affect the accuracy and completeness of the data. For example, some instructors may use the LMS grade book more extensively than others, which may result in differences in the amount and quality of data available for analysis. This variability may limit the ability to make meaningful comparisons across different courses and may make it difficult to draw conclusions about the effectiveness of specific interventions.

This study also does not account for other potential factors that may impact student success, such as mental health, socioeconomic status, or family obligations. For example, a student’s mental health status or financial situation may impact their ability to succeed in a course, even if they have a strong academic record. This limitation may limit the accuracy and completeness of the predictive model and may make it difficult to design effective interventions that address the root causes of student attrition.

Finally, it should be noted that the study excludes institutional policies, which may also be factors that contribute to students not being able to complete college in a timely manner, from institutional financial aid and transfer policies (Baum & Scott-Clayton, 2013; Monaghan & Attewell, 2015) to limited course availability and scheduling issues (Abele, 2021). Despite these limitations, this study provides a valuable foundation for future research and interventions aimed at improving graduation rates by identifying at-risk students early and providing targeted support.

Results

The first step in the research effort was to determine the model to utilize in the identification of students at risk of failing a course. Four models were developed and compared: logistic regression, random forest, kNN, and neural network. The Generalized Linear Model (GLM) logistic regression model output indicated that missing grades in the third week were not statistically significant (p > .05); as such, the variable was dropped, and the model was retrained. The final logistic model showed that the grade at week 5 (W05) and missing work in weeks 4 and 5 were all statistically significant (p < .05). The ‘Ranger’ random forest model implementation (Breiman, 2001) was trained and initial metrics were captured, followed by tuning the model across a 10-fold grid with the number of trees (100–1000) and a minimum number of nodes (5–50) as hyperparameters. The tuning of the hyperparameters indicated that a tree count of 300 and minimum node size of 50 was preferred using accuracy and receiver operating characteristic (ROC) curve/area under the curve (AUC) as metrics. The initial kNN model run used a nearest neighbor k-value of 5 and offered relatively poor results. Tuning on a grid of 10 k-values from 100 to 1000 resulted in k = 400, providing the best predictive results. Finally, the NNet model (Venables & Ripley, 2002, Chapter 7) was evaluated as a feed-forward neural network with a single hidden layer. The default model generated a 4-5-1 network with 31 weights and 100 epochs. The tuning process utilized a 6-level grid of epochs (50–200) and hidden units (3–24) and suggested a 4-15-1 network with 91 weights and 50 epochs.

Several relevant metrics were calculated for each of the models, including accuracy, sensitivity, precision, and specificity. Accuracy provides a ratio of all correctly predicted values to the total number of predicted values. Sensitivity, often referred to as the true positive rate, is the ratio of students predicted to fail a course to the number that actually failed. Specificity, commonly referred to as the true negative rate, reflects the ratio of students predicted to pass a course to the number who actually passed. ROC/AUC calculations provide a suitable means of evaluating model accuracy (Bowers & Zhou, 2019). The ROC/AUC values represent the area under the curve of the models’ receiver operating characteristic curves. The ROC curve provides an evaluation of model performance at all thresholds, and a higher AUC value signifies an increased ability of a model to better distinguish between passing and failing students.

Accuracy, sensitivity, precision, specificity, and ROC/AUC values were computed for the GLM logistic, random forest, kNN nearest neighbor, and the NNet neural net model (see Table 1). Values derived from 10-fold cross-validation indicated that the models shared very similar metrics, especially in terms of overall accuracy, with all four models having an accuracy score of 88%. Except for the GLM, the ROC/AUC scores were close to 81% for all models. The models were 97% correct in predicting students passing courses (specificity). The neural net model scored highest in sensitivity, with a 26.6% correct prediction rate of students failing courses (sensitivity).

Given that the research’s purpose was to explore a model’s ability to identify at-risk students, sensitivity values were of particular interest. The logistic regression and neural net models scored highest with correct predictions, 25.4% and 26.6% of the time, respectively, in terms of students who would fail in a specific course.

Table 1. Model Metrics for Models Compared in the Study Using 10-fold Cross-Validation and Spring 2022 Data

Note. ROC = receiver operating characteristic; AUC = area under the curve. Please refer to article text for information on models presented.

Although the GLM bested NNet in precision, the NNet model scored higher on ROC/AUC. As the GLM and NNet models produced the greatest number of true positives on the test data set, these two models were tested further. Data were pulled from the Fall 2021 semester, and the models were run using it for additional testing. A 10-fold cross-validation was performed, and the accuracy of the models was consistent with the initial results. The NNet model performed better in sensitivity, precision, and ROC/AUC, and the GLM scored higher in specificity. The metrics used for comparison are provided in Table 2.

Table 2. Comparison of Model Metrics Using Fall 2021 Test Data

Note. ROC = receiver operating characteristic; AUC = area under the curve. Please refer to article text for information on models presented.

An evaluation of the effect that changes in the MGT had in the models indicated that as the threshold increased, the accuracy, precision, and specificity increased with both the GLM and NNet models. Again, 10-fold cross-validation was utilized to prevent possible data-split bias. The evaluation found that sensitivity, a measure of particular interest in this effort, decreased as MGT increased in the GLM but increased to a maximum value at 80% MGT and then decreased at 90%. The ROC/AUC values peaked in both models’ 75% and 80% MGT ranges, reflecting the rapid drop in sensitivity despite increasing specificity. These values are presented in Table 3.

Table 3. MGT Effects on GLM and NNet Model Metrics Using k-fold Cross-Validation

Finally, N = 618 students in the Fall 2021 data set were identified as being at risk of failing at least one class. Of these, N = 171 students were identified as being generally at risk (students predicted to fail at least two courses). Seventy-four percent of these generally at-risk students ended up failing at least one class.

Discussion

The ability to detect at-risk students as early as possible—earlier than the once-traditional benchmark of mid-term grades—is crucial. It allows institutions to construct effective, proactive, and holistic intervention strategies to improve student success and retention.

We have demonstrated that it is possible to create a neural net-based model using easily attainable data from a widely used LMS system that can identify at-risk students—students who are predicted to fail at least one course—with reasonable accuracy as early as week 5. Our model performs as well as, and in some cases better than, other recently published models (Akçapınar et al., 2019; Baron et al., 2022; Cardona et al., 2020; Essa & Ayad, 2012; Gašević et al., 2016; Marbouti et al., 2016; Papamitsiou & Economides, 2014; Trivedi, 2022; Wolff et al., 2013) and by using, in most cases, less and more easily accessible data, is novel in its use of incorporating a simple missing grade flag indicating missed work for weeks 3, 4, and 5.

Detecting students at risk of failing a course earlier than week 5 proved difficult due to the lack of grade data available before week 5 (see Figure 1). We note that the dearth of early grade data before week 5 only reinforces the need to encourage faculty to follow the best practices of providing students with early low-stakes assessments and timely feedback (Brown et al., 2014). The week 5 mark does not seem unique to our institution, as week 5 is the suggested time for early detection in other works (Akçapınar et al., 2019; Cardona et al., 2020; Marbouti et al., 2016).

The identification of generally at-risk students by flagging students who are predicted to fail more than one course allows for focused non-academic intervention strategies to be carried out. The significance of the measure is evident, as 74% (N = 126) of students who were flagged as being generally at risk ended up failing at least one course.

Having two groups of at-risk students—students who have difficulty in only one course and students who have difficulty in multiple courses—allows for just-in-time academic support (tutoring, peer support, etc.) tailored individually for the former and more holistic intervention strategies that focus on the psychosocial well-being of students (success coach mentoring, one-on-one counseling, etc.) for the latter.

Future developments to refine the models discussed in this article could include, among other things, the addition of socioeconomic and demographic factors such as race, gender, residency, or status as a freshman, transfer, adult, or first-generation student to help identify at-risk students that have traditionally been identified as having more significant performance gaps.

Conclusion

The issue of identifying at-risk students in higher education is a major concern for all stakeholders, and it is likely to remain so for the foreseeable future. To help address this challenge, we have developed several models that can be used to identify students who are at risk of failing a course with an impressive 80% accuracy. These models rely on data that is readily available from a standard LMS system, making them easy to implement in most educational settings.

Specifically, our evaluations have shown that the most effective way to identify at-risk students is to use D2L Brightspace course grade data at week 5. By using a neural net model with a 4-15-1 network and 91 weights, and training it over 50 epochs, we were able to achieve an accuracy of 0.877, sensitivity of 0.265, precision of 0.601, specificity of 0.973, and ROC of 0.808. This model uses parameters that represent the student’s current standing in the course at week 5, as well as any missing grades or assignments flagged at weeks 3, 4, and 5, with an MGT threshold of 80%.

Importantly, we found that our generally at risk measure is also highly useful for identifying students who are likely to struggle in multiple courses. Our analysis revealed that 74% of students who were predicted to fail more than one course actually ended up failing at least one course. This highlights the potential of our approach to identify at-risk students early on, which can facilitate early intervention and increase the likelihood of student success and retention.

Overall, our findings suggest that our models and methods can help educators identify at-risk students in higher education with a high degree of accuracy and precision. By using these tools to detect students who may be struggling early on, educators can take proactive steps to provide targeted support and resources to help those students succeed. This could have significant benefits for both the students and the institutions they attend, ultimately contributing to better outcomes for everyone involved.

Author Note: We have no conflict of interest to disclose.

References