Undergraduate Internships, Degree Completion, and the Matthew Effect

The Internship Program and its Context

The internship program evaluated here focuses on public service, placing participants in community-based organizations (CBOs) and government agencies, doing work related to civic, economic, and environmental sustainability. Examples of work undertaken by these interns include assisting with climate-resiliency infrastructure projects; assisting seniors with technology use; community-based public health outreach; designing promotional materials for youth job training; youth education at a zoo; working with families on financial literacy; administrative support at a hospital; and research projects on energy efficiency. The university highly structures the relationship between interns and their employers. There is an involved application process for potential internship host sites (the employers), to ensure alignment with the program focus on community service. Employers then go through a training and orientation process run by the university.

Employers participate in a process, alongside students and their university supervisors, to create learning plans, then monitor the progress of these plans during and after the internship, including performance evaluation. Relationships between the employers and the university are strengthened through continued employer participation and hiring of former interns. Employers, as well as students, are assessed by the university, to ensure they are training and supporting students and furthering the goals of the program. There is a “job fair” process, where student applicants learn about and interact with representatives from the different host employers, and vice-versa. Students and employers express their preferences through ranked lists, with a high percentage of mutual matches. Student payroll and HR logistics are handled by the university.

Internships begin in the fall semester and continue into the spring semester, lasting 24 weeks in total. For this research we studied students who started their internship in fall semesters between 2013 and 2017. Interns work 11–12 hours per week, and were paid $12 an hour from 2013–2016, and $13 an hour in 2017. Some interns are eligible to earn college credits. The internship program is funded by the university and was explicitly designed as an alternative to the type of paid part-time work that these students often held outside of school. Paying program participants was conceived as a way to draw pragmatic students into an experiential learning environment focused on community service work, which would align with community service-oriented careers after graduation.

Beyond matching students with employers or community partners, the intern program provides regular campus-based training and development. Students participate in pre-service training, and monthly trainings during their internship, as well as receiving peer-to-peer support from other program participants. The program targets outcomes such as gains in workplace abilities, increased awareness of civic and social issues, strengthening of social and professional networks, and increased academic motivation.

Eligible students must be enrolled full-time in the spring before the program begins, and through the two semesters of the program. Applicants are expected to have a cumulative GPA of 2.5 before starting the program. Those from 4-year campuses must plan to complete 24 credits by the summer before the program begins, while 2-year applicants must have completed 18 credits. Prior to starting the internship, participants must also have completed any remedial education requirements, and, for most (but not all) participating campuses, must also qualify for Federal Work Study. These requirements, however, only make up a small part of the application process, which also included written essays, faculty recommendations, and, in some cases, interviews. These qualitative components asked the applicants to demonstrate enthusiasm and an orientation towards community service. As we discuss in our Findings section, university data indicated not all interns in reality met the explicit program requirements. This was because program officers had some discretion to waive certain requirements, based on the qualitative components of the application process.

Institutional Data

Our analyses used de-identified/anonymized and restricted-access administrative data on all students who attended the university from 2008 on. The core of that dataset was longitudinal, tracking every student from college entry until they graduated or left the university. For those who left the university without completing a degree, institutional researchers had also obtained data from the National Student Clearinghouse that documented subsequent degree completion from a college or university elsewhere.

This administrative data included student demographics—gender, ethnicity/race, age at entry, degree program and college, financial aid recipiency, and a measure of low-income status—and semester-by-semester and cumulative measures on enrollment, credits, and GPA. For students who entered the system’s community colleges, the data added assessments of remedial need in reading, writing, and mathematics. Students who participated in the service internship were flagged in the data, along with the dates of their internship. By prior arrangement with the state’s department of labor, university researchers had obtained quarter-by-quarter data for all students with earnings reported to the state’s unemployment insurance system.

Three variables served as the academic outcomes of primary interest: whether a student received either an Associate’s or Baccalaureate degree (from any college) within 16 semesters of college entry; the cumulative number of credits earned within 16 semesters of entry; and whether a student had left the university before completing a degree. Since students in the internship program typically began their placement at the start of their second year of college, we used students’ academic record measured at the end of their first year of college as covariates/controls in models assessing the association between having an internship and longer-term academic outcomes. We dropped the very small number of students who started their internship before the end of their first year.

Our analysis sample included all first-time freshmen who entered the university from 2008 on. We limited the analyses of graduation to cohorts with at least 16 semesters of possible attendance. We excluded Native American students from our analyses because of their small numbers. Descriptive statistics are reported in Online Appendix Table A1. (Due to length of material, the Online Appendix resides on the CUNY Academic Commons at https://www.inequalityinhighered.org/matthew-effect.)

Methods for Estimating Treatment Effects

The primary challenge in estimating the causal effect of an intervention or treatment such as the internship program is selection (Gelman et al. 2020; Imbens & Rubin, 2015). Not all students choose to participate (self-selection), or alternatively, program staff may include some applicants and exclude others based on the applicant’s academic record or other factors. Selection opens the possibility that the persons who were treated may differ in demographic background, prior academic achievement, or on other attributes from persons who were not treated. These factors are potential confounders. A simple comparison of the mean outcome for the treated with the mean outcome for the untreated would combine the effects of selection on confounders with those of the treatment itself, creating selection bias. Adjusting for selection and confounding is necessary to obtain a more accurate estimate of the Average Treatment Effect on the Treated (ATT).

Adjusting for selection is, unfortunately, non-trivial. Fundamentally, inferring causality from observational (rather than experimental) data is about comparing “like with like.” We want to compare, for example, the type of student who would self-select into the internship with another student of a very similar type who did not participate. One way of stating the problem is that we have to “get the model right.” By this, we mean we need to properly understand the relationship between our observed variables (confounders) and a student’s propensity to select into treatment. One approach is through modeling who gets the treatment, but we also attempt non-model-dependent methods for finding this relationship. Below, we use three methods designed to control for selection.

Coming at the problem from multiple directions, we are able to explore useful methodologies, while increasing our confidence in the robustness of our results. Inverse Probability-Weighting with Regression Adjustment (IPWRA) is a model-based method, while Coarsened Exact Matching (CEM) is an algorithmic, non-parametric method. In contrast with these two frequentist methods, we also explore a Bayesian machine learning method, Bayesian Additive Regression Trees (BART). Unlike IPWRA and CEM, BART models the relationship between confounders and outcome (the “response surface”). BART also, crucially, allows us to explore the heterogeneity of treatment effects, as these effects relate to the subgroups defined by the confounders.

IPWRA and CEM

Inverse Probability-Weighting with Regression Adjustment (Wooldridge, 2002, 2007, 2010) uses a two-stage method to estimate average treatment effects. The first stage models the treatment, using inverse-probability-weighting (IPW), while the second stage models the outcome, using regression adjustment (RA). This process is used to estimate the “potential outcomes” (Imbens & Rubin 2015), or counterfactual values, for treated and untreated students, and the ATT is the difference between these “potential outcome means” (POMs). The combined IPWRA has a “doubly robust” quality, which means that if the parametric form of either the treatment or the outcome is modeled correctly, one obtains an unbiased estimate of the ATT (Tan, 2010).

Coarsened Exact Matching (Blackwell et al., 2009; Iacus et al., 2008, 2009) is a non-model-based method of causal inference. Unlike IPWRA, CEM uses an algorithm to create a series of strata, each defined by unique combinations of the confounder values. The number of treated and untreated cases in each stratum is used to construct weights. Balance is measured before and after these weights using the ℒ₁ statistic developed in Iacus et al. (2008). Continuous covariates are “coarsened” (the C in CEM) and transformed into categorical variables.

Because each stratum is defined by the interaction of all confounders, this largely side-steps the problem of finding a correct parametric model for the treatment. Unlike a model-based approach to balancing data, Blackwell et al. (2009) point out that the CEM procedure “bound[s] the maximum imbalance” (p. 528) through the choice of coarsening bins, either by the researcher, using subject-area expertise, or automatic coarsening. For the results presented below, we used outcome models with all covariates included in addition to CEM weights, but results were similar to non-model runs which included only the treatment variable and CEM weights. Iacus et al. (2009, 2011, 2012) and King et al. (2011) have shown the advantages of CEM over other matching methods in terms of, “its ability to reduce imbalance, model dependence, estimation error, bias, variance, and other criteria,” (Iacus et al., 2012, p. 2) which they demonstrated using multiple real and simulated data sets.

BART

The third method we use for estimating the effects of the internship on student outcomes is BART (Chipman et al., 2006; Chipman et al., 2010; Hill et al. 2020; Hill et al., 2023). Methods such as IPWRA assume that the relationship between the confounders, treatment, and outcomes are linear, and that treatment effects are consistent across all individuals (homogenous). These assumptions can be unrealistic and may result in biased estimates when dealing with complex, real-world data. BART addresses these limitations by using regression trees (Hastie et al., 2009) to capture intricate, nonlinear interactions between confounders and outcomes. Regression trees flexibly model the outcome using subgroups defined by combinations of confounders. However, regression trees may “overfit” data, making the model difficult to generalize. The use of “boosting” (Hastie et al., 2009), a process which iteratively fits and combines a series of regression trees, can improve accuracy while reducing overfitting. Bayesian priors set “principled” (Hill et al., 2023, p. 424) constraints on these processes, keeping the complexity of the trees within reasonable bounds.

Again, untreated control students are used to estimate the potential outcomes for the treated students, and vice versa. However, in the Bayesian framework, individual potential outcomes are modeled for each student. BART estimates the individual treatment effect for each student as the difference between these potential outcomes. There is quite a bit of uncertainty in these individual estimates, so BART takes averages of them. These model-based estimates are conditional on all the confounders (including interactions of confounders) used to fit the regression trees, so these are called “individual conditional ATTs” (iCATTs). Averaged iCATTs are called Conditional Average Treatment Effects on the Treated (CATTs).

There are major advantages to using the Bayesian framework in causal analysis when estimating treatment effect heterogeneity (Carnegie et al., 2019; Green & Kern, 2012). By summarizing the distributions of iCATTs by subgroup, such as between male and female students, one can estimate and compare CATTs for these groups. Traditional methods of estimating heterogenous treatment effects involved selecting interaction terms and estimating treatment effects for subsets of the data. Manually choosing interactions creates all sorts of complications, and using data subsets can dramatically shrink N’s. In contrast, the BART process identifies interactions using boosted regression trees set within Bayesian priors. All final estimates are conditioned on these covariate interactions, meaning that subgroup CATTs are essentially estimated using the full model and data. Finally, unlike frequentist methods, comparisons of this kind in a Bayesian framework do not create issues of statistical significance based on repeated comparisons, which crop up when choosing among possible interactions and data subsets. The advantages of BART for causal inference have been demonstrated through a series of contests, hosted by the Atlantic Causal Inference Conference (Dorie et al., 2019; Thal & Finucane, 2023). See Online Appendix E for more on BART’s performance in these contests.

Bayesian Credible Intervals

Because the BART method is set in a Bayesian framework, we assess statistical significance using “credible intervals,” rather than confidence intervals and p-values (Kaplan & Little, 2024). The BART process estimates a posterior distribution of likely values for parameters. Unlike distributions, such as the normal distribution, which can be described using two parameters, these posterior distributions are generated through the modeling process, are complex, and cannot be easily described. However, we can summarize these posterior distributions using the mean of the distribution and the 95% credible interval. This credible interval has a straightforward interpretation, as it means there is a 95% probability that the parameter value falls within it, based on our prior assumptions, model, and data. If the 95% credible interval does not overlap zero, we report the parameter estimate as statistically significant. One advantage of the 95% credible interval is that it pairs nicely with the standard 95% confidence interval used in frequentist research, even though the interpretation is different.

Sensitivity Analysis

To confirm the robustness of our results, we also have run two types of sensitivity analysis, “Robustness of an Inference to Replacement” (RIR; Frank et al., 2013), and “Impact Threshold of a Confounding Variable” (ITCV; Frank, 2000), using the ‘konfound’ package in R (Narvaiz et al., 2024). The results of these tests gave us strong confidence in the validity of our results. Discussion of this sensitivity analysis methods, and results, can be found in Online Appendix F.

Who Became an Intern?

Online Appendix Table B1 reports two logistic regression models predicting enrollment as an intern (i.e., participation in the treatment) among two-year Associate’s students (N = 84,782). The first model has a single predictor: a race-gender variable. The second model adds covariates describing students’ academic progress in their first year of college and other traits prior to taking the internship.

The first noteworthy finding is that women, especially women of color, were considerably over-represented in this internship program, relative to their overall numbers at these colleges. Asian women had an odds ratio of 3.67, Hispanic women an odds ratio of 2.66 and Black women an odds ratio of 3.75, compared to the reference category—White men. Black men (odds ratio 2.50) and Asian men (3.014) were also overrepresented relative to White males in Associate’s programs, but to a lesser extent than women. There was a non-significant difference between Hispanic men and White men.

The addition of other covariates capturing students’ pre-intern academic performance reduces the size of the odds ratios for the race-gender categories, but the general pattern remains: Women of color disproportionately enrolled as interns. The logistic regressions also indicated that academically stronger students—those who earned more credits in their first year of college and had higher GPAs—were significantly more likely to enroll in the internship program. In addition, there was a clear link between students’ earnings in their first year of college and their subsequent likelihood of becoming an intern. Students who did not have any paid employment in their first year of college were significantly less likely to intern. Individuals who earned between zero and $2,000 in their first year of college had an odds ratio of 1.58, indicating they were more likely than non-earners (the reference category) to become interns. Conversely, undergraduates who earned over $10,000 in their first year of college were much less likely to take an internship in the subsequent year, with an odds ratio of 0.59. At both extremes—non-earners and high earners—the odds of interning were lower. A possible explanation is that because these were paid internships, they appealed more to students who needed some income than to students who had managed their first year of college without working for pay. But since the internship’s pay was modest (about $2,500 a semester) it had less of an appeal to undergraduates who already earned more than that from employment during their first year of college. There was also a statistically significant odds ratio of 1.59 for those students from low-income backgrounds, indicating they were more likely to participate than those from non-low-income backgrounds.

Our finding that students with stronger academic records in their first year—in terms of GPA and credits accrued—were also more likely to become interns, net of other factors, reflects institutional barriers to entry. On its website the internship program indicated undergraduate GPA and credit requirements for applicants. In practice we can tell from university records that not all interns met those “requirements”; they were more of a desideratum than a strict GPA cutoff, and were taken into account alongside qualitative factors.

Online Appendix Table B2 provides equivalent models predicting who entered an internship, but for those undergraduates in four-year Baccalaureate programs. The pattern was similar to Associate’s students in terms of race-gender, except that the difference between Hispanic and White men at this level was statistically significant, with Hispanic men having an odds ratio of 2.04 without covariates and 1.65 with covariates. Internship participation levels for those with a better academic track record, and those from low-income backgrounds, were similar to Associate’s students.

What Were the Average Effects of Interning?

Table 1 reports the ATTs from the internship for three different outcomes: degree completion within 16 semesters from college entry, dropping out (leaving college without completing a degree) within those 16 semesters, and the number of credits accumulated within 16 semesters. In each case, the ATTs were estimated separately for students in Associate’s and Baccalaureate degree programs.

The columns in Table 1 report ATTs using three different methods that adjust for selection into the internship: CEM, IPWRA, and BART. We have discussed the rationale for each method in an earlier section of this paper.

The ATTs for degree completion were consistent across the three models (CEM, IPWRA, and BART): in each case treatment effects indicated beneficial effects, were statistically significant, and of substantial magnitude. For example, for the Associate’s degree completion outcome, ATTs were 0.167, 0.178. and 0.159, meaning that after controls those participating in an internship had a 16.7 percentage point greater probability (or 17.8 percentage point or 15.9 percentage point) of completing a degree within 16 semesters, than non-interns. The negative signs on the Associate’s outcome for dropping out of college without completing a degree imply that interns were 13.4 percentage points lower in their dropout rate (or 16.4 percentage points or 13.9 percentage points lower in the other two models) than those who did not intern.

Note. † Bayesian credible interval does not overlap zero, meaning a greater than 95% probability of an effect. Degree and Dropout ATT’s represent percentage point changes, e.g., 0.167 means the treated group (interns) had a 16.7 percentage points greater probability of the outcome, relative to similar students in the untreated group. CEM = Coarsened Exact Matching; IPWRA = Inverse-Probability-Weighted Regression Adjustment; BART = Bayesian Additive Regression Trees. ATT’s and uncertainty measures for the BART models are “population” estimates (PATTs), based on the posterior predictive distributions for both potential outcomes. Standard errors are based on the standard deviations of these posterior predictive distributions.

*p < .05. **p < .01. ***p < .001.

Table 1 documents that similar ATTs were found for both students in Associate’s programs and those in four-year Baccalaureate programs. Having found substantial treatment effects on average, the next stage in the analysis was to examine treatment effect heterogeneity: Did different types of students benefit to the same or different extents from the internship treatment?

Treatment Effect Heterogeneity

We used our BART model to explore the effects of the treatment on different subgroups defined by socioeconomic difference and pre-internship academic performance. BART estimates a treatment effect for each individual, conditional on all the covariates in the full model. ATTs can then be estimated for subgroups by simply summarizing the distribution of these individual treatment effects for members of that subgroup (Carnegie et al., 2019). Because these ATTs are conditioned on the covariates, they are referred to as CATTs. One of the many benefits of fitting a Bayesian model, which estimates treatment effects for each individual student, is that we are able to summarize these individual CATTs (iCATTs) in many ways, while avoiding the issues of multiple comparisons which arise in frequentist models.

Our analyses address heterogeneity of benefits from the internship in two ways. First, we estimated CATTs for different subgroups to determine whether each subgroup received a beneficial and statistically significant CATT on the three outcomes. These are presented in Online Appendix Tables C1-C4. Next, we looked at differences between the subgroups and estimated whether these differences were statistically significant, as reported in Online Appendix Tables D1-D4.

Online Appendix Tables C1-C4 show that all subgroups of students—men and women, Whites and minoritized groups, low-income and non-low-income, higher and lower GPA students—had beneficial outcomes from the internship, measured as higher degree completion, a lower likelihood of dropping out, and a greater completion of credits. These CATTs were statistically significant and beneficial for all the subgroups.

This procedure indicated that internships benefited men more than women, with statistically significant differences for all outcomes except number of credits earned for Associate’s students. Low GPA undergraduates got more of a boost from an internship than higher GPA students. The lowest GPA category, a first-year GPA between 2.0 and 2.5, had a significantly higher effect from the internship than the average, and the highest GPA category (3.5–4.0) had a significantly lower effect than the average, across all outcomes for Associate and Baccalaureate students, except number of credits earned for Associate’s students. Students with a 2.5 to 3.0 GPA generally got a larger than average effect. Finally, students with a 3.0 to 3.5 GPA were close to average, with no statistically significant difference between the average internship effect and the effects for these students across all outcomes but Baccalaureate credits earned.

For race-gender categories, the CATT differences were less clear or consistent. Among Associate’s students, Black men had significantly higher CATTs from an internship than the grand mean CATT for all race-gender categories when it came to the effects on degree completion and for protection against dropout. Among Baccalaureate students, both Black and Hispanic men had larger effect sizes than the grand mean. Most other contrasts were either not significant or were inconsistent across outcomes. However, Asian and White women had significantly lower CATTs than average for some outcomes.

Modeling Probability of Completion

Part of the BART process is fitting two boosted-tree models that predict a given outcome: one for the treated students and one for the controls. Based on these modeled predictions, we can classify students into tertiles, having a low, medium, or high probability of completion—which we define as completing a degree or not dropping out in 16 semesters—as if they did not participate in the internship, and then summarize the individual CATTs (iCATTs) for the students in these subgroups.

Based on this classification, low, medium, and high completion probability groups all had beneficial and statistically significant CATTs from an internship on both the graduation and drop-out outcomes, as can be seen in Online Appendix Table C5.

One notable finding was that students with a low probability of completion, based on their demographic and pre-internship academic performance, received on average a significantly larger benefit from the internship (a larger CATT) than students who had a high likelihood of degree completion. Some of these differences were substantial, as can be seen in Table 2. For the degree completion in 16 semesters outcome, students with the lowest probability of completion had a 12.8 percentage point higher CATT than the grand mean (15.9 percentage points) at the Associate’s level, and a 16.3 percentage point higher CATT than the grand mean (16.9 percentage points) at the Baccalaureate level—nearly double the average beneficial effect.

To reiterate: we observed significant beneficial effects of the internship—statistically significant CATTs—across all subgroups. But the effect sizes were significantly larger for men, for Black and Hispanic men in particular, for students with a low first-year GPA, and for low likelihood of completion students, than for other subgroups. This has implications for practice and policy that we consider below.

Discussion and Conclusion

This paper uses quantitative evidence on undergraduate academic performance and outcomes to compare several thousand undergraduates who participated in a year-long paid internship program at several two- and four-year campuses of a large urban university, with otherwise similar undergraduates who did not participate in the internship. In most cases, participants took this internship in their second or third year of college. The internship program was distinctive in that the work placements were in community-based, non-profit, and government organizations, not at private corporations. The internships were paid to make sure that students who needed to earn enough to cover basic living expenses were able to participate. There was a strong emphasis on service—that by participating in these internships students were contributing to their communities.

Using three different methodological approaches to estimating treatment effects, each of which provides a different corrective to selection bias, we found that interning had positive academic spillover effects, improving participants’ chances of retention and graduation, and reducing the likelihood that a student would subsequently leave the university without a credential. Positive spillovers were documented for all subgroups of students: men and women, minority and non-minority, high and low GPA students, Associate’s and Baccalaureate students, etc.

Note. To determine whether there is a difference between the CATTs of completion groups and the average treatment effect for all students, we subtract the distribution of CATTs for each completion group from the grand mean distribution of CATTs. If the credibility interval does not overlap zero, this means there is a greater than 95% probability of a difference between the group CATTs and the grand mean CATT. Degree and Dropout contrasts represent percentage point differences, e.g., 0.128 means students in this completion group had a 12.8 percentage points higher CATT than the grand mean treatment effect for all students.

However, we found that those students in the lowest tertile likelihood to complete received a substantially larger benefit from the internship than those with a high likelihood to complete. Further, subgroups which, on average at this university, have a lower likelihood of degree completion, such as students whose academic performance in the first year of college (pre-internship) was not strong, and Black and Hispanic men, received larger internship benefits than other subgroups.

Limitations of our research have to do with both methodology and the structure of the internship program. For any causal analysis of observational data, controlling for selection is key, which means controlling for confounding variables. The challenge is knowing whether or not we have controlled for all relevant observed confounders, and worse, if there are important unobserved confounders. Our sensitivity analyses, discussed in Online Appendix F, give us confidence in our results. The second major limitation has to do the implementation of the internship program across sites: whether each intern was really getting the same treatment. Heterogeneity in how the different internship programs at different institutions interacted with different employers creates the potential for a heterogeneity of treatments. Two things give us confidence in this regard. First, this was a particularly centralized and structured program, which should enforce a level of isomorphism, especially as employers participate year after year. Second, because we are measuring average treatment effects, implementation heterogeneity is largely averaged out, and any treatment effect heterogeneity we examine is not based on heterogeneity between sites.

Our research addresses a key question in the literature on the effects of college internships on academic outcomes: whether additional extracurricular time spent on internships comes at the expense of academic pursuits (Binder et al., 2015; Prescott et al., 2021; Routon & Walker, 2019). We find that this internship, with a structured relationship between universities, students, and employers, which is paid, and focused on community service, has positive academic benefits. However, our main finding addresses the heterogeneity of these effects. Many other researchers have examined how internship effects vary by student characteristics, such as race, gender, family income background, and academic preparation (e.g., Duarte et al., 2023; Hora et al., 2019; Routon & Walker, 2019). However, the BART method gives us greater confidence that we have properly modeled the relationship between these characteristics and the treatment effects, and further, allows us to explore treatment effect heterogeneity using a variable modeled from these characteristics—probability of completion.

These findings on heterogeneous effects call into question current practices around recruitment into internships. It is understandable that colleges want to recruit academically successful students for internships because interns represent the university to the outside world, to community groups, non-profits, and government agencies and the university wants to “put its best foot forward.” But paradoxically, we found that the beneficial effects of these service internships were greater for less academically successful students, for those at higher risk of non-completion, and among Black and Hispanic male students, whose dropout rates tend to be high.

The policy implication of our findings is that universities should view community- and service-based internships as a valuable intervention for improving academic progress and completion precisely for those undergraduates who are currently least likely to succeed in college. Instead of treating internships as an opportunity for the strongest students, colleges should identify those students at highest risk of non-completion and recruit and encourage those students to participate in such a service internship program. This focus would undoubtedly require additional resources for outreach and pre-internship training for at-risk students, but our findings suggest that this strategy would likely pay off in terms of degree completion for those students at highest risk of non-completion.