Early Indicators of Student Success: A Multi-state Analysis

Data

This project linked researchers from four states who were interested in participating in a joint project, and who had access to and experience with their state’s postsecondary data systems (SURDS). Each state team undertook separate data analyses based on their own state’s SURDS. For security and legal reasons, no data was shared or moved across states; instead, each state team followed a common protocol for statistical analyses using their own state’s data. Only statistical outputs were shared. The selection of the four states—Illinois, New York, Texas, and Virginia—was not random or representative in any way. Each had a well-established SURDS and researchers willing to participate. This unusual collaboration was funded by the Bill & Melinda Gates Foundation, in part to establish whether this kind of multi-state collaborative research was practical and useful.

The researchers analyzed undergraduate cohorts that entered college from Fall 1999 through Spring 2010 and were followed through 2016. In three states, data were analyzed separately for public four-year colleges and for two-year community colleges. For one state (Illinois) only data on community colleges were available. Although data on private college enrollments were available in some SURDS, private colleges did not usually report transcript information, so we excluded them from the analyses reported in this paper. The SURDS were longitudinal, following each student from entry into college for at least ten years, at which time indicating whether an individual had graduated. Sample sizes ranged from 48,783 Bachelor of Arts (BA) students in Texas, to 357,836 for Associate in Arts (AA) students in New York.

Variables

Graduation was constructed as a binary outcome variable. For students initially entering a four-year college, we defined our milestone as completion of the bachelor’s degree within 12 semesters of entry. The equivalent milestone for undergraduates who started at community colleges was more complicated. Students who begin at community college often say they intend to earn their baccalaureate, and many transfer to a four-year college without first completing their associate degree (Long & Kurlaender, 2009). Thus, simply counting whether a student received an associate degree is misleading as a measure of community college student success. Instead, we counted as having reached an important milestone those community college matriculants who either obtained an associate or baccalaureate degree or had accumulated 60 or more credits, which is the minimum required at most community colleges to receive an associate degree.

After exploratory modeling, we settled on the following independent variables available at the beginning of each student’s freshman year: age at college entry; parental adjusted gross income (AGI); high school GPA, SAT, ACT, or Texas Assessment of Knowledge and Skills (TAKS) score; remedial requirements (math, reading, writing); workload in the first semester (total number of credits, counting both remedial and non-remedial courses); and whether a major was declared at college entry (a dichotomous variable). Another set of variables contained measures of student performance during the first semester of college: GPA in first semester; credits earned in first semester; whether remedial math was taken in the first semester (if required), and if so whether it was passed; and equivalent measures for taking or passing remedial reading or remedial writing in the first semester. All continuous variables were converted into categorical predictors, allowing for the addition of a ‘missing’ category for each variable.

Table 1 reports details for each variable. There were some differences in data availability between states. For example, Texas used a statewide assessment of math and English skills that is mandatory for its high school seniors, transforming this into a percentile score, whereas the New York data used SAT scores. Different state SURDS also had somewhat different measures of low income. Some used eligibility for free or subsidized school lunches in high school, while others used Pell eligibility or adjusted parental income. Consequently, the variables in models are not identical across the states, although they are quite similar. We also examined whether adding measures that described student performance after the first year of college improved the accuracy of models predicting graduation. We found that later academic performance measures did not substantially improve prediction. Combined with SURDS items, data on how well a student performed during their first semester of college were sufficiently predictive, as documented below. The specific logistic regression equations are available in an online appendix using the following link: https://inequalityinhighered.org/2019/12/03/early-indicators-of-student-success-a-multi-state-analysis/.

Table 1. Variables Description (New York)

Avoiding Algorithmic Bias

Readers will note that we deliberately eschew using gender, race, or ethnicity as predictors of graduation in the statistical models that follow. Despite being aware that these demographic characteristics are on average associated with higher or lower completion, we avoided building models predicting graduation that relied on group characteristics of this type. To do so might reify stereotypes and lead to what economists term ‘statistical discrimination’—assessing individuals’ promise by their group membership, resulting in disadvantages for those individuals whose performance differs from their group average (Arrow, 1973). More recently, the potential for bias in computerized decision-making has been conceptualized as ‘Algorithmic Bias’ (Baer, 2019; Baker & Hawn, 2021; Government Accountability Office, 2022; Noble, 2018; O’Neil, 2016). The core idea is that ostensibly fair or neutral computer programs used to make decisions may nevertheless contain features in their algorithms which can systematically disadvantage one group of persons compared to another. In computerized decisions about bail, for example, Black defendants have been shown to have a higher error rate of being incorrectly classified as likely to reoffend and therefore are more likely than Whites to be denied bail (Corbett-Davies et al., 2017). Baker and Hawn (2021) provide a review of the terminology, concepts, and literature regarding algorithmic bias in education.

Given these concerns regarding algorithmic bias, we constructed models that omit gender, race, and ethnicity as predictors of graduation. As a check, after completing our analyses without such attributes, we determined separately for each state’s SURDS whether the inclusion of gender and race/ethnicity would have improved predictive accuracy. Adding those variables made very little if any improvement in predictive accuracy, given the behavioral measures already in the model. There were two demographic exceptions: a student’s age at college entry and a family income measure were both associated with graduation such that omitting those predictors would impair predictive power. We judged that incorporating those two variables into the models would be less problematic than building predictive models in which gender, race, or ethnicity played a substantial role.

Statistical Models

We followed a multi-stage strategy for creating and evaluating predictive models. Initially, we randomly assigned the undergraduate sample into two different parts: training and test samples (Rogers & Girolami, 2012). Our training data, consisting of 70% of the randomly selected cases, was used to construct a predictive logistic regression model. The remaining 30% of the full sample, called test data, was withheld from the logistic regression and used in a final step to evaluate the predictive accuracy of the regression model.

After a logistic regression model was estimated from the training data, the prediction equation obtained from that analysis was applied to score the cases in the unused test sample, providing a predicted probability (or p-hat) of reaching the graduation milestone for each individual in the test group. We standardized the distribution of p-hats into decile groups of equal size. To evaluate the model’s accuracy in its tails, we focused on the bottom of this distribution: those cases for which our model indicated a very low chance of graduation. The predicted probabilities calculated from our model were then compared (cross-tabulated) with the actual measured milestone outcomes for the test-sample individuals, yielding validation statistics that measured the accuracy of our predictions for the test sample.

Cross-validation assesses whether a model developed from a training sample generalizes to out-of-sample test data, reflecting the overall population, and therefore can be considered reproducible. Cross-validation is a protection from what data miners term ‘overfitting’—the possibility that a strongly predictive model partly reflects random noise or finds relationships in a dataset that would not apply to data drawn from other samples (Rogers & Girolami, 2012). In all our tables presented below, the reported accuracy statistics are always for the held-back test data, which is the equivalent of applying the predictive model to new data.

The Logic of Prioritizing Intervention and Support Services

The existence of a murky middle does not lessen the value of the early indicator models if the purpose of intervening is to enhance graduation rates. Knowing which among their incoming students are most and least likely to graduate, provided in a timely fashion by our early indicator models in the form of risk scores, would allow college staff to prioritize outreach to students and target support services to those at greatest need. In our view, prioritization of interventions and support services is the most immediate and practical use of our early indictor models of student success.

It is possible that some institutions might use predictive scores to separate matriculants at highest risk of non-completion from other students at less risk, and tailor a special program for the former. One analogous situation is the City University of New York’s pre-matriculation program, CUNY Start. This voluntary program identifies applicants to community college who have multiple remediation needs according to placement-test scores in math, reading, and writing, taken just before they intend to start college (City University of New York [CUNY], 2017), only a fraction of the data we use in our multivariate predictive models. Identified students at CUNY are invited to defer immediate enrollment in a community college program, and instead are offered the option of taking one or two twelve-week courses that focus on remedial coursework, at very low tuition, taught by teachers especially skilled at adult education (CUNY, 2017). The goal of this program is to raise students’ skills to such a level that they can pass the skills tests and begin their community college program without further remedial coursework.

Initial evaluations of the CUNY Start program report that significantly larger proportions of Start students pass the skills tests needed to exit remediation than a comparison group of community college students who take remedial coursework alongside non-remedial classes, during their early semesters at community college (Scrivener & Logue, 2016). Some students who do not pass their courses in this special track may decide not to enter community college—we do not have data on how many—and if so, would have paid far less tuition than they would have done had they started community college and taken remedial coursework there. One should recall that Adelman (2006) found that about 13% of entering students in a national sample drop out before completing ten credits.

A second policy option would use risk scores to identify high-risk students in order to offer those students additional academic or social supports (cf. Tinto, 2012). Another analogue can be found in the City University of New York’s Accelerated Study in Associate Programs (ASAP), which identifies potential participants based on only a fraction of the data we utilize: low placement-test scores that cause students to take one or two remedial courses. ASAP offers selected participants a range of extra supports, such as individual advisors or counsellors whom they meet on a regular basis, block schedules, tracking into courses with other similarly situated students, and material benefits like free textbooks and transportation. CUNY’s ASAP program is a complex intervention, initially targeted at high-risk community college students. Random assignment evaluations have documented near-doubling of graduation rates for ASAP students compared to control groups (Gupta, 2017).

In 2015, Ohio began replicating this ASAP program in three of its community colleges. Student volunteers were randomly assigned treatment of academic services similar to CUNY’s, as well as financial assistance in the form of tuition and textbook waivers, and career advisement. Ohio’s ASAP program doubled the graduation rate for students with developmental requirements and significantly boosted graduation rates for those without. This program cost the colleges an additional 42% per student but cost 22% less per degree conferred compared with the control group (Miller et al., 2020). These two programs illustrate two types of policies where participants are selected using early or leading indicators of student success. We stress, however, that these two real-world examples did not use the multivariate indicators discussed earlier in this paper, instead selecting participants on a voluntary basis, based on the placement test scores of incoming undergraduates.

This brings us to the important issue of how error in prediction might affect such policies regarding interventions. The first thing to note is that the current indicators of risk being used by many colleges, specifically skills or placement test scores from commercially available ACCUPLACER or COMPASS tests, have been criticized for being inaccurate (Rodriguez et al., 2014; Scott-Clayton, 2012). The status quo approach to identifying at-risk students and directing them into special classes is already error laden.

For the higher-risk end of the spectrum, most predictions of non-graduation in the models we developed were 95% accurate or better, meaning that at most five in a hundred identified as being at risk of non-completion would in fact have completed their degree. One ethical issue is whether individuals (or colleges) would be harmed if these 5% were erroneously classified as high-risk. If risk scores are used to prioritize provision of extra academic and counselling support, it seems unlikely that a misclassified student would be harmed by being encouraged to make use of such targeted support, especially since students may decide to spurn those supports if they so choose. From the institution’s perspective, if they identify students for extra support, 95% of whom would not be likely to complete their degree and (due to inaccurate prediction) 5% would graduate anyway, then perhaps 5% of the extra supports are wasted in the sense that they would better be targeted elsewhere. In our judgement, this is a relatively small misallocation of resources, with little risk of harm to students who were misclassified as needing those supports. More serious harm would occur if risk scores were used to discourage students deemed at-risk from attempting a degree program, since about 5% of those identified persons would have completed a degree. We would argue against that type of use for early indicators.

There are additional rationales for using early indicators to target institutional interventions and support. In the past, counseling, tutoring, and other academic support services have often followed a ‘first-come, first-serve’ approach, leaving it to each student facing difficulties to come and ask for help. Unfortunately, many students who need help do not seek it. Researchers have shown, for example, that young Black men tend not to make full use of academic support systems (Bush & Bush, 2010). Both early alert systems and the SURDS approach suggested by this paper take a proactive or intrusive approach to academic support: the institution reaches out to specific students who seem to be facing difficulties. That model focuses resources on those most in need and prioritizes outreach and intrusive advisement over waiting for students to seek help.

This research collaboration has shown that it is possible to develop early indicators of student success and non-completion and to link those early indicators to interventions. As more states provide access to SURDS data, we expect to see wider use of early indicators of student success as a way of targeting support services and for assessing institutional interventions aimed at improving student success.