Unlocking Undergraduate Student Success: A Study of High-Impact Practices in a Comprehensive and Diverse College

Context of the Study

The study took place at a comprehensive public four-year teaching college that places a special emphasis on the advancement of a diverse and largely underserved student population. As both a Minority-Serving Institution (MSI) and Hispanic-Serving Institution (HSI) designated in 2013 and 2015, respectively, a large portion of the enrolled students in this college were female (75%), came from racially minoritized groups (70% non-White), or were first-generation (47%) based on the enrollment census data in fall 2019. Over one-third of the students (38%) were low-income, Pell Grant recipients. Although its position as an MSI and HSI with a large racially minoritized, first-generation, and low-income student population makes this Institution unique among teaching colleges, diverse students like these are a growing portion of college-going students nationally (Espinosa et al., 2019).

Data Sources and Participants

Two types of data were utilized for the study: senior students’ NSSE data from 2015 through 2019 and institutional administrative data. We retrieved NSSE data through the designated interface for this Institution. The Institution has participated in NSSE every year since 2015. Each spring, a specific online survey link was sent directly to eligible first-year and senior-year students enrolled in the Institution. One of this paper’s authors downloaded the NSSE data and linked them to the institutional data warehouse through student IDs collected by NSSE and then transferred the de-identified data to another co-author to perform the data analysis. Both the NSSE and institutional data allowed us to examine outcome variables related to student experiences with HIPs, engagement indicators, perceived gains, overall satisfaction, and other institutional success metrics measured by persistence and graduation.

From spring 2015 through spring 2019, 1,673 senior students completed the NSSE core survey. Of these students, 1,482 responded to the questions related to HIPs and were included in the data analysis. The majority of the participants were female (79%), came from a racially minoritized group (48%), were first-generation (65%), and had received a Pell Grant at any point during their time as a student at the Institution (53%). The sample of NSSE seniors for this study was similar to the campus senior population enrolled in fall 2014 through fall 2018 (76% female, 55% racially minoritized groups, 64% first-generation, and 55% Pell Grant recipients). Though relatively representative of the larger student body on campus, as with all studies using a sample from one institution, caution should be applied when interpreting and generalizing the findings of this study.

Measures and Variables

We used the NSSE core survey to measure senior students’ participation in HIPs, engagement indicators, perceived gains, and overall satisfaction. Comparable questions in the NSSE surveys of 2015 through 2019 allowed us to combine and analyze data from multiple years.

HIPs. The HIP questions, unlike other NSSE questions, were not limited to the current school year at the time of data collection; therefore, seniors’ responses reflect their participation in any/all the years they attended the Institution. One question asked students to indicate their participation in six HIPs, “Which of the following have you done or do you plan to do before you graduate?” HIPs in NSSE included: (a) an internship, co-op, field experience, student teaching, or clinical placement; (b) a learning community or some other formal program where groups of students take two or more classes together; (c) taking courses involving a community-based project (service-learning); (d) working with a faculty member on a research project; (e) a study abroad program; and (f) a culminating senior experience (e.g., capstone course, senior project or thesis, comprehensive exam, portfolio). Except for the service-learning item, each HIP had four possible responses: (a) have not decided, (b) do not plan to do, (c) plan to do, and (d) done or in progress. We recoded “d” as “1,” indicating HIP participation, and “a,” “b,” and “c” as “0” for no HIP participation. The service-learning question asked, “About how many of your courses at this institution have included a community-based project (service-learning)?,” and the four possible responses were (a) none, (b) some, (c) most, and (d) all. We recoded “a” as “0” for no service-learning in the courses, and “b,” “c,” and “d” as “1” for courses including service-learning opportunities.

To measure students’ combined participation across these six HIPs as a whole, NSSE also created a new scale that ranged from “0,” meaning that a student had not participated in any of the identified HIPs, to “6,” indicating a student who reported participating in all six practices. Accordingly, we developed a coding system aligning with the scale to measure HIPs: (a) qualitatively, participated or not; (b) quantitatively, cumulative number of participations of the six HIPs (0–6); and (c) categorically, range of participation (0 HIPs, 1–2 HIPs, 3–4 HIPs, and 5–6 HIPs).

Student engagement was measured by 10 engagement indicators constructed with 47 questions in NSSE. The prompt asked, “During the current school year, about how often have you done the following?” The question was followed by a list of activities, such as “given a course presentation,” each with four response options: 1 = “never,” 2 = “sometimes,” 3 = “often,” and 4 = “very often.”

Perceived gains measure students’ self-reported gains in practical competence, personal and social development, and general education competency areas as a result of their undergraduate education. The perceived gains questions asked, “How much has your experience at this institution contributed to your knowledge, skills, and personal development in the following areas?” Ten areas included, for example, “working effectively with others” and “acquiring job- or work-related knowledge and skills.” Each area has four response options: 1 = “very little,” 2 = “some,” 3 = “quite a bit,” and 4 = “very much.”

Overall satisfaction was measured by two NSSE questions. One question asked, “How would you evaluate your entire educational experience at this institution?” and the other one asked, “If you could start over again, would you go to the same institution that you are attending?” Both questions had four response options: 1 = “poor,” 2 = “fair,” 3 = “good,” and 4 = “excellent” for the first question, and 1 = “definitely no,” 2 = “probably no,” 3 = “probably yes,” and 4 = “definitely yes” for the second question.

To facilitate comparisons over time and between groups of NSSE participants within or across institutions, NSSE recommended converting scores of engagement indicators, perceived gains, and overall satisfaction on a 60-point scale (NSSE, 2013). Responses to these questions were recoded with values of 0, 20, 40, or 60, for 1, 2, 3, or 4, respectively. Engagement indicators were pre-calculated by NSSE. We recoded each question on perceived gains and satisfaction and computed the composite scores for perceived gains and overall satisfaction using the SPSS syntax developed by NSSE (NSSE, 2013). Higher scores indicate more frequent engagement, higher-level perceived gains, and more satisfaction.

Exploratory variables in this study included self-reported gender, race/ethnicity, first-generation status, and institution-recorded low-income status. We grouped participants into five major subgroups based on race/ethnicity categories: Asian, Black or African American, Hispanic of any race, two or more races, and White. Although it is worthy to explore other racial and ethnic groups, we combined them as “other” due to a small sample size of each. We focused our analysis on five groups: Asian, Black, Hispanic, two or more races, and White. First-generation students are those with neither parent having graduated with a bachelor’s degree from a four-year college or university. Low-income students are those who have ever received a Pell Grant based primarily on the student’s or parents’ income for the previous year (Wei & Horn, 2002).

Additionally, persistence or graduation status as the outcome variable was drawn from institutional data. We used persistence as one of the indicators of senior student success, measured by a student either having graduated by the summer after completing the NSSE survey or continuing enrollment in the Institution. A student either having graduated by summer from the Institution or re-enrolled next fall in the Institution was coded as “1” for “persisted,” otherwise, the student was coded as “0” for “not persisted.” Similarly, graduation status was defined by whether a student had graduated from the Institution by the summer after completing the NSSE survey and was coded as “1” for “graduated” or “0” for “not graduated.”

Data Analysis

To address the research questions, we adopted a quantitative method with several analytic approaches. Before conducting data analysis, we examined if the data warranted univariate or multivariate analysis of variance (MANOVA). Descriptive statistical analyses were performed to address the first research question. We compared the overall participation rate in HIPs and each HIP based on students’ race/ethnicity, first-generation status, and low-income status.

To address the second research question, we conducted the General Linear Model (GLM) with two MANOVAs and one univariate analysis of variance with Bonferroni corrections in each case, examining the mean differences in engagement indicators, perceived gains, and overall satisfaction by HIP participation and race/ethnicity, first-generation status, and low-income status since multiple composite scores of engagement indicators and perceived gains were examined as well as one overall satisfaction score. NSSE recommended that the proportion of respondents within a given demographic variable, for example, gender or full-time/part-time students, might differ substantially from their population percentages, or that students within a subgroup might differ substantially in the variables of interest (NSSE, 2014). Therefore, weights (e.g., gender or full-time/part-time students) have to be used to calculate weighted means and standard deviations for engagement indicators, perceived gains, and overall satisfaction scores (NSSE, 2014). The GLM, allowing the inclusion of sampling weights, was utilized for the calculation (Chen et al., 2009; NSSE, 2014).

To examine the interaction effects between HIP participation and race/ethnicity, HIP participation and first-generation status, and HIP participation and low-income status, we first performed three two-way analyses of variance for engagement indicators, perceived gains, and overall satisfaction, respectively. The results revealed that there were no significant interaction effects between HIP participation and race/ethnicity, first-generation status, and low-income status. Therefore, the results section mainly focused on the main effects of HIP participation on student outcomes.

Due to the categorical nature of persistence and graduation status (“persisted” or “graduated”), we conducted the chi-square test to address the third research question. The chi-square test can be used for testing dependence or homogeneity (Franke et al., 2012) and was used to examine the proportional differences in persistence or graduation status between HIP participants and HIP non-participants based on race/ethnicity, first-generation status, and low-income status.

To address the fourth research question, a hierarchical logistic regression was performed to test for the association between HIP participation and persistence by adding control variables of gender, race/ethnicity (five groups), first-generation status, and low-income status as a block to the model. Then we added HIP participation status to the model. These two blocks of variables were entered into the predictive equation in a hierarchical order to examine which variables significantly predict the outcome variable of persistence with an additional block of variables introduced. The chi-square test was used to examine whether the model was statistically significant, and the Wald test was used to determine statistical significance for each of the predictive variables. We used female, White, non-first-generation, and non-low-income as the reference groups (coded as “0,” respectively) for the regression model. For the test, the alpha level for statistical significance was set at 0.05.

Limitations

We acknowledge several limitations of this study. First, institutions, as well as their students, choose to participate in NSSE (neither institutions nor students are randomly selected). Propensity score matching or other matching strategies may reduce this selection bias; however, these matching strategies may need a larger sample size. Future studies may enlarge the sample size and/or use matching strategies to investigate these research questions further. Second, this study relies on self-reported measures of HIPs and engagement, perceived gains, and satisfaction. Validated measures of students’ actual participation were not analyzed, warranting further research. Future studies may include data from other sources and/or collect more qualitative data to triangulate with quantitative measures (Finley & McNair, 2013). Third, although the data for this study were collected from college seniors, the participation in HIPs did not necessarily happen only in the senior year. However, this study provides an informative snapshot of student experience with HIPs by senior year. Lastly, the results should not be used to make a causal inference about the relationship between HIP participation and student outcomes due to the cross-sectional design of this study.

HIP Participation Patterns

The results of data analysis yielded several salient findings. Descriptive statistical analysis showed that, overall, 77% (n = 1,144) of senior students reported that they had participated in at least one HIP. Among the six HIPs measured in NSSE, seniors participated in service-learning the most (62%), followed by an internship, co-op, field experience, student teaching, or clinical placement (34%), a culminating senior experience (24%), a learning community (17%), undergraduate research with a faculty member (13%), and study abroad (3%). Of the students who participated in HIPs, more than half of them participated in just one or two HIPs.

Table 1 summarizes the proportion and number of students participating in HIPs by race/ethnicity, first-generation status, and low-income status. On average, senior students have participated in just one or two types of HIPs.

Rates of participation in at least one of the six HIPs were similar among Black (75.5%), Hispanic (77.8%), two or more races (75.3%), and White students (76.9%), and slightly higher for Asian students (80.0%). However, the participation rates in “faculty research” and “culminating senor experience” for Asian students were lower (6.5% and 16.1%) relative to students from other racial and ethnic groups. The HIP participation rates were similar for first-generation (77.6%) and non-first-generation students (76.5%). Similarly, no significant equity gaps were identified regarding participation in each type of HIP between racially minoritized groups and first-generation students and their advantaged peers. There were no significant mean differences by race/ethnicity or first-generation status regarding the average number of HIP categories in which students participated. Overall, a significantly larger proportion of low-income students (82.0%) participated in HIPs relative to their non-low-income peers (71.4%). This held true for each of the six HIPs (see Table 1). In addition, low-income students participated in more types of HIPs than their non-low-income counterparts (M = 1.68 vs. M = 1.36).

Table 1. HIP Participation by Race/Ethnicity, First-Generation Status, and Low-Income Status

HIP Participation and Engagement, Perceived Gains, and Overall Satisfaction

HIP Participation and Persistence (Including Graduation)

Regarding institutional outcomes, overall, we observed an increase in persistence for HIP participants versus HIP non-participants. For seniors who participated in HIPs, a significantly larger proportion graduated by the summer after completing the NSSE survey or re-enrolled the next fall (93.0%) compared to their peers who did not participate in HIPs (76.3%), χ²(1) = 75.34, p < .01, phi = 0.23. The overall persistence boost was 16.70% for HIP participants against HIP non-participants (see Figure 4). This trend was true for all subgroups examined, including race/ethnicity, first-generation, and low-income students. Black, two or more races, first-generation, and non-low-income HIP participants gained a higher boost than their peer counterparts (see Figure 4).

Regarding graduation, we observed an increase in the graduation rate for HIP participants versus HIP non-participants. For seniors who participated in HIPs, a significantly larger proportion graduated by the summer after completing the NSSE survey (45.1%) compared to their peers who did not participate in HIPs (15.7%), χ²(1) = 95.50, p < .01, phi = 0.25. The overall graduation boost was 29.4% for HIP participants against HIP non-participants (see Figure 5). This trend was true for all subgroups examined. Asian, Hispanic, two or more races, non-first-generation, and low-income HIP participants gained a higher boost in graduation than their peer counterparts (see Figure 5).

Logistic Regression Results of Persistence

We performed a logistic regression of persistence on HIP participation controlling for gender, race/ethnicity, first-generation status, and low-income status. The hierarchical linear regression analysis showed that by adding two blocks of variables to the predictive model, the model was statistically significant, χ²(8) = 99.14, p < .01. After controlling for other variables, the results indicated that HIP participation was a significant predictor of persistence, Wald F(1) = 59.50, p < .01 (see Table 2). This indicates that there is a positive relationship between HIP participation and persistence. For a student who participated in HIPs, the odds of this student persisting would increase by 304%. Holding other variables constant, low-income status was a significant predictor of persistence, Wald F(1) = 10.95, p < .01. The odds of a low-income student persisting would decrease by 46%. Holding other variables constant, overall, race/ethnicity was also a significant predictor of persistence, Wald F(1) = 23.95, p < .01. Examining this effect closely, we found that relative to White students, the odds of persistence for being an Asian, Black, and two or more races student decreased by 27%, 68%, and 41%, with Wald F(1) = 1.29, 16.68, and 1.90, respectively. However, this reduction was significant higher for Black students, p < .01. Interestingly, holding other variables constant, relative to White students the odds of a Hispanic student persisting would increase by 46%, although this increase was not significant, p > .05. Gender and first-generation status were not significant predictors of persistence, Wald F(1) = 2.11 and 0.53, p’s > .05, respectively (see Table 2).

Table 2. Results of Logistic Regression of Persistence Status

Note. ** p < .01.

HIP Participation Patterns

The findings of this study show that a large proportion of seniors (77%) at this Institution have participated in HIPs, but typically just one or two in their entire collegiate career. This low level of participation is generally in alignment with findings from Finley and McNair’s (2013) study of more than 25,000 undergraduates, though we found greater similarity in participation rates among first-generation and non-first-generation students (1.56 compared to 1.48, respectively) compared to their findings (1.24 compared to 1.45, respectively). Among the six types of HIPs, senior students reported participating more in service-learning, internships, and culminating senior experiences than the other three HIPs. This is likely due, at least in part, to the embedding of these particular kinds of HIPs in the curriculum and requiring them for course/program completion. Contrary to findings of Finley and McNair, overall, there were no equity gaps in terms of HIP participation by race/ethnicity and first-generation status, implying equal access to these opportunities that may reflect a lessening of barriers over time or in this particular institutional context. However, we do find that Asian students reported participating in “faculty research” and “culminating senor experience” at a relatively lower rate than their other racial and ethnic peers. A disproportionately high number of Asian students are enrolled in pre-nursing or nursing programs, where undergraduate research is not as common as in other disciplines. Therefore, this might explain the reason why Asian students reported less participation in research activities. Another reason may be that Asian students do not participate to the same degree as their peers in co-curricular programs on an HSI campus, such as Nepantla and the Teacher Academy Pipeline Project (TAPP), which provide pathways that connect students with research opportunities. Future research on the aforementioned findings is worth further exploration.

Interestingly, this study reveals that low-income seniors participated in more HIPs (and more types of HIPs) than their advantaged peers. We hypothesize this finding is largely associated with institutional student success initiatives designed for low-income students, particularly those of Hispanic descent (as the Institution is an HSI and MSI). Other potential institutional factors that could have influenced this finding might include the ongoing culture of holistic student development, a recent curriculum redesign that encourages HIPs, first-year experience programming with strategic faculty hires, and aligned co-curricular activities. These efforts were aimed at using evidence-based approaches for improving students’ success, especially for students who are low-income, first-generation, and/or underprepared (Tucker et al., 2020). There is also a possibility that some other cultural (non-institutional) factor drives low-income students to participate in HIPs at a higher rate, or perhaps non-low-income students participate more in experiences outside of the Institution (e.g., travel, hobbies, internships) that leads to decreased utilization of HIP opportunities within the college. This is another area for future study.

HIP Participation and Engagement, Perceived Gains, and Overall Satisfaction

The current study indicates that relative to their peers who have not participated in any HIPs, students who have participated in HIPs reported higher levels of engagement, perceived gains, and overall satisfaction. The top three engagement indicators reported were: higher-order learning, discussions with diverse others, and quality of interactions, all known to relate positively to student success (Johnson et al., 2016; Kuh et al., 2006; Webber et al., 2013). Importantly, these are skills students need to be successful during and beyond college and in the workplace (Carnevale & Smith, 2013, 2018; Craig & Bridges, 2005; Silberstein & BrckaLorenz, 2019). Similarly, both advantaged and disadvantaged students reported higher levels of perceived gains with increased participation in HIPs (Figure 3), which confirms the findings of Finley and McNair (2013) and Kuh (2008). Therefore, more institutionally supported mechanisms and culturally engaging environments to facilitate participation in an increased number of HIPs should be explored (required capstones, first-year learning communities, etc.). Our data also suggested HIPs are effective ways of engaging all students, particularly racially minoritized students, a population we strive to retain and graduate at higher levels in higher education (Museus, 2014; O’Donnell et al., 2015). Lastly, seniors who participated in HIPs reported more overall satisfaction with their college experience than did their counterparts who did not participate in HIPs, a finding consistent in extant literature (Kinzie, 2012; Kuh et al., 2017).

HIP Participation and Persistence or Graduation

More encouragingly, we do observe the higher boosts in persistence and graduation rates for HIP participants compared to HIP non-participants. The boosts in persistence rates were higher for Black, two or more races, first-generation, and non-low-income HIP participants. Furthermore, the boosts in graduation rates were higher for Asian, Hispanic, two or more races, non-first-generation, and low-income HIP participants. The findings of the study also reveal that HIP participation was significantly associated with increased persistence and graduation, especially for racially minoritized populations (Black, Hispanic, and two or more races) and low-income students. Although teaching-focused in nature, the Institution strategically creates a culturally engaging environment and offers curricular and co-curricular opportunities for students to conduct research projects with faculty and closely interact with faculty and their fellow peers. This practice was found to be a powerful way of enhancing senior students’ persistence or graduation (Cresiski et al., 2021; Museus, 2014).

The current study also indicates that HIP participation was a significant predictor of persistence or graduation status, corroborating findings from similar studies where the impact of HIP participation was examined (D’Souza et al., 2018; Nosaka & Novak, 2014). If a student participated in HIPs, the odds of this student being retained or graduating would increase by 304%. Low-income status was also a significant predictor of persistence or graduation; the odds of persisting would decrease by 46% for low-income students relative to their non-Pell Grant peers. However, we observed a larger gain in graduation for low-income HIP participants than HIP non-participants. Again, we hypothesize that the role of specific institutional student success initiatives designed for low-income students may be a factor as those programs have been shown to significantly enhance persistence and graduation rates. In addition, financial aid opportunities and the availability of student support services coupled with work-study opportunities and varied course formats designed to support disadvantaged students at this Institution likely contributed to the greater number of low-income students persisting and graduating compared to their non-low-income peers. Millea and colleagues (2018) found similar results. This support most likely allowed low-income students the opportunity to focus on school as opposed to having to divide their time between school and other work obligations, which is typical for many low-income college students (Carnevale & Smith, 2018).

Given the positive association between HIP participation and enhanced student success metrics affirmed in this study, we developed three recommendations for implementing HIPs in similar institutions:

1. 1) Embed or require more HIPs within a curriculum to increase access to and participation in these opportunities. HIP usage can even be embedded at a systemic level, for example, in program review. Faculty could be incentivized to assign HIPs as part of their course requirements. Disciplines that do not traditionally require internships in the community or faculty-mentored research could partner with other disciplines that do to develop unique cross-disciplinary HIP opportunities (Leach, 2016).
2. 2) Encourage faculty consideration of HIPs in course development or revision by requiring faculty to identify which, if any, HIPs will be included in the new or revised course (this could be part of the form submitted to the curriculum committee, for example).
3. 3) Make the messaging for HIPs inviting for all students. Some students might perceive HIP messaging on recruitment materials as services targeted solely for remedial or developmental purposes and they will not want to be identified in this manner.

Ultimately, it is essential for every institution to intentionally create more opportunities for students to experience HIPs to enhance their engagement, perceived gains, satisfaction, and academic success.