Impact Without Randomization: Lessons From a National, Longitudinal Evaluation

Abstract

Higher education professionals often are tasked with providing evidence to stakeholders that programs, services, and practices implemented on their campuses contribute to student success. Furthermore, in the absence of a solid base of evidence related to effective practices, higher education researchers and practitioners are left questioning what works—and what does not—on college campuses. The goal of this essay is to advance a conversation about evidence within the higher education literature through an illustration of a nonrandom research study designed to evaluate the impact of a national interfaith organization on institutions’ efforts to promote interfaith learning and development. We briefly provide an overview of the relationship between evidence and evaluation efforts, then move to illustrating one project as a vehicle for understanding how evidence emerging from a nonrandom evaluation design can be done effectively in a national study related to one dimension of student success, and finally, offer support for the essay’s central thesis—that evaluation designs can present reliable evidence of impact in the absence of randomization.