Empirical Benchmarks for Planning and Interpreting Causal Effects of Community College Interventions

Studies and Analysis Samples

Our analyses focus on evaluations of CC interventions where the identification strategy allows for an unbiased estimator of intervention effects. In doing so, we ensure that the distribution of effects is not confounded with cross-study variation in degrees of bias. In addition, we aimed to identify evaluations that are representative of the CC interventions that have been rigorously evaluated to date, to optimize the comprehensiveness and generalizability of the benchmarks.

Accordingly, the findings in this paper are based on 30 well-executed RCTs of postsecondary interventions conducted by MDRC, which represent all but one of the postsecondary RCTs that MDRC had led from 2003–2019 (the one RCT excluded from the present analysis had limited follow-up). We present findings on student outcomes for the first six semesters after random assignment, a common time to consider CC degree completion rates, thereby making it possible to examine the pattern of variation in effect sizes across semesters of follow-up as well as across interventions.

Importantly, the impact findings from these RCTs are causally robust. Twenty-seven of these RCTs have been reviewed by the U.S. Department of Education’s WWC and have all met the WWC’s evidence standards without reservations. The three RCTs that have not yet been reviewed almost certainly meet the same standards, given their similar design, analytic approach, and attrition rates.

Furthermore, the RCTs used for this paper comprise a sizable portion of all large-scale postsecondary RCTs conducted in the United States. Only 68 large-scale (with more than 350 participants) postsecondary RCTs have been reviewed and met WWC evidence standards without reservations. Thus, the findings from this paper likely provide a representative picture of the distribution of effect sizes in well-executed CC RCTs.6

Some of the 30 RCTs used in this paper are multi-arm trials. These multi-arm trials evaluate the effect of more than one intervention in a single RCT by, for example, randomly assigning students to a control group, intervention A, or intervention B. Thus, although there were 30 RCTs in our sample, they are used to estimate the distribution of effects of 39 interventions.7 The resulting full study sample includes 39 postsecondary interventions and a total of 65,604 students.

As shown in Table 1, the 39 studied interventions vary in their key components (e.g., advising, tutoring, financial supports, etc.) and duration (from one semester to three years). For more information on the key components of each individual intervention, see Appendix Tables A2 and A3. For even greater detail, Appendix Table A1 provides links to original reports about each intervention.

The eligible population in most studied interventions was students enrolled in the colleges (as opposed to prospective students or applicants); in two thirds of interventions, eligibility was limited to new or first-year students (see Table 2). Most interventions were implemented at CCs or public universities with large populations of students from families with low-income (or the interventions targeted students from families with low-income) and most studies included multiple cohorts. The interventions were implemented across 44 postsecondary institutions (mostly CCs) and 12 states.

Note. One intervention was financial aid reform that did not result in any increase in the amount of aid distributed. It is therefore the only intervention with none of the seven intervention components that were coded. Sources: MDRC calculations using data from THE-RCT and reports and journal articles. A list of reports and articles can be found in Appendix Table A1.

Note. Interventions are equally weighted. Sources: MDRC calculations using data from THE-RCT and reports and journal articles. A list of reports and articles can be found in Appendix Table A1. 

a Percentages do not add up to 100% because interventions may have more than one eligibility criteria.

Reflecting national patterns in two-year colleges, most students (77%) in the average study are younger than 25 (see Table 3). Almost two thirds of students (60%) in the average study are female, and the average percent Black is 25%, the average percent Hispanic is 36%; both percentages are higher than in the average two-year college in the United States. There is substantial variation in the characteristics of students across the studies; for example, the percentage of female students ranges from 0%–92%, and the percentage of White students ranges from 0%–60%.

Note. Interventions are equally weighted. Sources: MDRC calculations using data from THE-RCT and reports and journal articles. A list of reports and articles can be found in Appendix Table A1. 

The analytic sample used to estimate the distribution of effects varies across outcomes and by semesters depending on data availability, ranging from 20 to 39 interventions, and 21,163 to 65,604 students.8 Table 4 shows the percentage of individuals and interventions that are included in the analysis of effects for each outcome and semester, relative to the full study sample. In semesters 1–3, at least 82% of studied interventions and students are included in the analysis; however, in semesters 4–6, the number of studied interventions with longer-term follow-up data drops. For example, about two thirds of studies (26 studied interventions) collected enrollment data in semester 6 and half of studies collected credits and degree completion data (22 and 20 studied interventions, respectively) through semester 6. For this reason, caution is needed when interpreting results for outcomes beyond semester 3. The distribution of effects is very likely upward biased due to follow-up selection bias. That is, interventions with more promising short-term impacts were more likely to have longer-term follow-up data (see Bailey & Weiss, 2022).

Note. The full sample includes 65,637 students and studies of 39 interventions. Sample sizes by outcome and semester are shown in Appendix Table A4.

Because the data used for the present analysis are from actual RCTs—nearly all of which focus on students who are already enrolled and who agreed to participate in the study—the distribution of effects presented in this paper may not generalize to the effects that one would observe if these interventions were offered to all CC students at the colleges (or in the United States), nor to prospective students or applicants to these colleges. However, the findings are likely to represent the range of effect sizes that researchers will encounter for the subset of colleges and enrolled students who are interested in these types of interventions.

Data Sources and Measures

Parameters

Before delving into how we estimate key parameters of interest from our data, we first define two key parameters of interest. Let B_j be the true average effect of intervention j. Our analyses begin by estimating two parameters that summarize the cross-intervention distribution of intervention mean effects—its mean (β) and its standard deviation (τ). By definition:

and

β and τ provide a summary of the central tendency and spread of the distribution of average true effects across interventions, respectively.

In addition to these two primary summary statistics, we aim to characterize the distribution of true effects by identifying points on the distribution that correspond with percentiles of the distribution.

Important Context about Distributions of Estimated Effects

Much of the K-12 education (and other fields) literature base on empirical benchmarks starts with a group of studies and the estimated effects of the interventions from those studies. Empirical benchmarks often include the mean, median, standard deviation, and various percentiles of the distribution of the estimated effects from those studies (for examples, see Bloom et al., 2008; Hill et al., 2008; Kraft, 2020; Lipsey et al., 2012). Notice the emphasis on estimated.

Define as an original estimate of the average effect of intervention j, estimated using an unbiased estimator (e.g., a simple difference-in-means estimator or a regression-based estimator). Such estimates () are a combination of the true effect (B_j) and random estimation error (r_j). That is:

Consequently, the spread (or variance) of the distribution of effect estimates is a combination of variation in the true effects of the interventions () plus independent variation in estimation error (). Accordingly, the distribution of effect estimates is expected to be wider (and sometimes much wider) than the distribution of true effects (for more details, see Bloom et al., 2017 and Hedges & Pigott, 2001). That is:

which can be re-written as:

Recall from Equation 2 that τ, the standard deviation of the cross-intervention distribution of true average effects, is a key parameter of interest. The standard deviation of , as is commonly presented in the literature, overestimates the spread of the distribution of true effects. Relatedly, percentiles of the distribution of estimated effects present an inaccurate depiction of percentiles on the distribution of true effects. We attempt to address this issue with our chosen estimators.

Estimators

To examine the distribution of true effects across studied interventions, we use the fixed-intercept, random treatment coefficient (FIRC) model described in detail by Bloom et al. (2017) for studying cross-site impact variation.14 The FIRC approach was used by Weiss et al. (2017) for their secondary analysis of data from 16 multi-site RCTs of education and training programs.

Specifically, we use the following 2-level hierarchical linear model to estimate β and τ:

Level 1: Sample Members

Level 2: Studied Interventions

where:

In this model, Y_ij is the value of the outcome (e.g., credits earned) for individual i in studied intervention j, S_ij is a vector of random assignment block indicators (one for each block in each studied intervention) set equal to one if student i in studied intervention j was randomly assigned in that random assignment block and zero otherwise, T_ij equals one if individual i in studied intervention j was assigned to treatment and zero otherwise. The blocks account for the fact that individuals were randomly assigned within blocks (e.g., colleges and cohorts) and that the proportion of sample members randomized to treatment can vary across blocks. The model does not control for students’ baseline characteristics (like their gender and age). Doing so would not appreciably improve the precision of estimated effects because available characteristics are only weakly correlated with outcomes (Somers et al., 2023), and very few baseline variables are consistently available across studies.

An important feature of the FIRC model, relevant to the purpose of this paper, is that it allows for intervention-specific effect coefficients (B_j) that can vary randomly across studied interventions. The B_j’s are modeled as representing a cross-intervention population distribution with a mean value of β and a standard deviation of τ. Hence, the intervention-level random error term, b_j, has a mean of zero and a standard deviation of τ. Critically, when estimating τ, FIRC accounts for estimation error associated with each intervention specific effect estimate.15 Bloom et al. (2017) provide further information about this model and Raudenbush and Bloom (2015) explore its properties.

For the present analysis, the FIRC model is fitted to the analysis samples separately for each outcome and semester. Estimates of β (average) and τ (standard deviation) are key summaries of findings.

To further aid with interpretation, we also provide percentiles of the distribution of intervention effects. As previously noted, because of estimation error a key challenge with calculating percentiles is that the distribution of estimated study-specific effects (e.g., the reported in the original studies) exaggerates the amount of true cross-study variation in effects (Bloom et al., 2017; Raudenbush & Bloom, 2015). We address this problem using a two-pronged approach proposed by Bloom et al. (2017). First, we begin by using the results of the FIRC model to compute the empirical Bayes shrinkage impact estimate for each intervention, , which is a weighted average of the intervention-specific average impact estimate, , and the overall average impact estimate, , where the weight of the study-specific estimate is based on its reliability, λ_j:16

This means that for small-sample studies, where estimated effects are estimated less reliably, the empirical Bayes estimates will be “shrunken” towards the grand mean impact estimate. The shrinkage factor is based on an estimate of reliability. The resulting distribution of empirical Bayes effect estimates varies less than the best estimate of the variance of the distribution of true effects (Raudenbush & Bryk, 2002, p. 88). That is, . Thus, the variance of empirical Bayes estimates will typically understate the cross-study variance of true mean program effects, and by extension, be smaller than the estimated cross-study variation from the FIRC model (). Hence, as a second step, we calculate an “adjusted” empirical Bayes estimate for each study to compensate for this over-shrinkage (Bloom et al., 2017):

where γ is an adjustment factor that stretches the distance between the empirical Bayes estimates and the mean impact estimate :

This adjustment inflates the variance of the empirical Bayes estimates to be exactly equal to the estimated variance of true program effects () from the FIRC model. The percentiles presented in this paper are based on the adjusted empirical Bayes estimates, . Study-specific estimates are also available in a public-use dataset created for this paper (available at https://www.mdrc.org/the-rct-empirical-benchmarks).

Distribution of Impact Estimates—An Example Using Three Estimators

Before presenting our main results, it is useful to examine the distribution of intervention impact estimates for a single outcome at a single time point, highlighting some important points about our methodological approach. Figure 1 presents the estimated impact of each intervention on the cumulative number of semesters enrolled through two semesters after random assignment. Impact estimates are presented using three estimators: (a) original impact estimates (called , above, and “OLS” in Figure 1)17, (b) empirical Bayes impact estimates (called , above), and (c) adjusted empirical Bayes impact estimates (called , above). Interventions are listed in the order of the magnitude of their adjusted empirical Bayes impact estimate. The horizontal axis of the figure indicates the direction and magnitude of each impact estimate.

First, notice that the spread of the is 16% larger than the spread of . Specifically, , whereas the . The latter, by construction, is equal to the estimate of τ, the standard deviation of the intervention-level distribution of average effects, from Equation 7. As shown in Equation 5, variation in is expected to be larger than τ, owing to estimation error. The aim to correct for this (and the slightly overcorrect for this in favor of other desirable statistical properties). Thus, the may yield the most accurate representation of the spread of the distribution of true effects.

This is particularly relevant when planning a study and considering the MDTE. If a planned study has a MDTE of 0.125 semesters enrolled, examining the distribution of estimated effects () would show that effects of at least 0.125 semesters enrolled have been observed in 5 out of 39 interventions. While a relatively large effect, researchers might argue that being in the top 12% of interventions seems feasible, depending on the intervention. Examining the distribution of adjusted empirical Bayes estimates () might make telling that story more challenging. The adjusted empirical Bayes estimates suggests that an effect of 0.125 semesters enrolled has only occurred in 3 out of 39 interventions, implying that the intervention proposed for study would need to have a true effect in the top 6% of interventions in THE-RCT for the proposed study to be adequately powered. This could change researcher or funder decision-making.

Next, notice that the rank order of interventions’ estimated effects occasionally changes, depending on the estimator. This is especially notable in evaluations where: (a) the original impact estimate is relatively more or less precise than the impact estimates for the other interventions (usually due to a relatively small/large sample size) and (b) the original impact estimate is far from the mean () impact estimate. One notable example is the EASE Info + $ Summer ’18 intervention. While is the sixth largest OLS impact estimate, is the fourth largest adjusted empirical Bayes impact estimate. Compared to the other evaluations at the top end of the distribution, EASE Info + $ Summer ’18 was a relatively large evaluation (total sample size around 3,500). Owing to the large sample size (and precise impact estimate), the difference between and is small and so EASE Info + $ Summer ’18’s adjusted empirical Bayes impact estimate rank is better than its original impact estimate rank.

Such rank switching is probably a good thing—estimation error and related issues probably ought to result in shrunken expectations about the true effects of interventions with relatively impressive results coming from trials with imprecise impact estimates. Making such adjustments when interpreting findings from a new trial is prudent. The smallest trial in THE-RCT included 444 students, with all others including over 700 students. If a new trial with 150 students finds an estimated effect of 0.15 cumulative semesters enrolled through two semesters after random assignment, our best estimate probably should not be that this intervention is more effective at increasing cumulative semesters enrolled than nearly all other interventions in THE-RCT. By shrinking expectations, we can help combat “the winner’s curse” and “promising trials bias” (Simpson, 2022; Sims et al., 2022). Appendix B provides a discussion of how to take the estimated effect of an intervention (and its associated standard error) and calculate an adjusted empirical Bayes impact estimate that can be located on the distribution of adjusted empirical Bayes impact estimates from this paper. An online tool associated with this paper allows users to make this adjustment easily, by plugging in an effect estimate and its standard error from a new RCT (go to https://www.mdrc.org/the-rct-empirical-benchmarks).

Figure 1 illustrated key points about our methodological approach using one outcome and time point as an example. We now turn to a broader discussion of the findings across major outcomes and time points.

Estimated Distribution of True Impacts

Appendix Table A4 presents information on the estimated distribution of true effects (mean, standard deviation, and percentiles) across the 39 postsecondary interventions in THE-RCT, by outcome and by semester. The first two outcomes (enrollment and credits earned) are marginal, focused only on what happened in that semester. The other three outcomes (cumulative semesters enrolled, cumulative credits earned, and degree earned) are cumulative, such that earlier impacts carry forward. As discussed earlier, the distribution of effects beyond semester three or four should be interpreted with caution because these distributions are based on fewer studies and very likely biased upwards because studies with more promising short-term effects are more likely to have longer-term follow-up data. Figure 2 provides a visual summary of Appendix Table A4 for four outcomes and for six semesters after random assignment.

Consider first the effect distribution on credits earned through one semester. The mean of the distribution (β) is 0.48 credits. This implies that, on average, the interventions in THE-RCT had positive impacts on students’ first semester credit accumulation. Next, is the estimate of true impact variation across interventions (τ). This cross-intervention standard deviation is estimated to be 0.55 credits. This estimate of τ is larger in magnitude than the mean impact (β), indicating substantial variation in the effectiveness of interventions under study on this outcome at this time point. Lastly, are estimates of the magnitude of effects at various points in the effect distribution—the 10th, 25th, 50th, 75th, and 90th percentiles are −0.03, 0.13, 0.48, 0.60, and 1.29, respectively.

Cutoffs to create rules of thumb for what is small, medium, or large are arbitrary, resulting in odd distinctions (e.g., an effect estimate of 0.59 credits might be considered medium and an effect estimate of 0.61 might be considered large, despite their being indistinguishable for practical or statistical reasons). Thus, it may be preferable to simply characterize the relative magnitude of an effect estimate in terms of its approximate percentile within the distribution of effects. Nevertheless, some people find rules of thumb helpful, so one might consider effects on credits earned in semester one below 0.13 (25th percentile) to be relatively small, between 0.13 and 0.60 credits (75th percentile) to be medium-sized, and above 0.60 credits to be relatively large.

When looking across outcomes and semesters, a few patterns emerge:

• • For marginal outcomes, downward trends for means, and decreasing variability over time.

The mean (across interventions) effect on marginal enrollment and credits earned decreases over time. For example, the mean effect on credits earned starts at 0.48 credits in semester 1, and decreases to 0.43, 0.25, 0.17, 0.05, and 0.02 credits in semesters 2–6, respectively. This downward trend holds for enrollment. Similarly, the spread of the intervention-level distribution of effects is widest in semester 1 and decreases over time. These trends suggest that a lot of the action, with respect to intervention effects, occurs early on. Consequently, in the short-term, an intervention’s effects on enrollment and credits earned must be larger in absolute magnitude to be considered large relative to the effects of other interventions.

• • For cumulative outcomes, upward trends, and increased variability over time.

The mean (across interventions) effect on cumulative number of semesters enrolled and cumulative credits earned increases over time. For example, the mean effect on cumulative credits earned starts at 0.47 credits in semester 1, and increases to 0.89, 1.14, 1.43, 1.70, and 2.46 credits, in semesters 2–6, respectively. This pattern holds for cumulative enrollment. Similarly, the spread of the intervention-level distribution of effects is smallest in semester 1 and increases over time. These trends show that the effects of some of the more effective interventions continue to grow throughout the first three years after random assignment. Consequently, over time, an intervention’s effects on cumulative enrollment and cumulative credits earned must be larger in absolute magnitude to be considered large relative to the effects of other interventions.

Because degree completion is a longer-term outcome, the number of studies with follow-up data is smaller than for other outcomes. Based on the subset of studies with available data (20 to 23 interventions, depending on the semester), the mean effect size is 0.9 percentage points by semester 4, 1.6 percentage points by semester 5, and 1.7 percentage points by semester 6. As noted earlier, these average effects are likely biased upwards due to follow-up selection bias. One notable aspect of the degree findings is that two outlier interventions drive a lot of the action. The original effect estimates from these studies are approximately 16 and 18 percentage points. No other study’s original effect estimate is above 4 percentage points. Thus, benchmarking degree impacts based on the evaluations in THE-RCT is limited. It seems reasonable to consider any positive effect on degree completion to be an impressive feat, with effects larger than 5 percentage points being notable.

Planning

The findings in this paper can be used to plan the sample size for future evaluations of CC interventions. For example, when planning a study of an intervention that is lighter-touch (perhaps a single-component intervention), researchers may want to choose a sample size that will make it possible to detect effects at the lower end of the distribution of effects presented in this paper (for example, 0.23 cumulative credits earned through one year). Whereas for more comprehensive interventions, which tend to have larger effects (Weiss & Bloom, 2022; Weiss et al., 2022), a larger minimum detectable true effect may be sufficient, depending on the goal of the study. Importantly, other design parameters are necessary to calculate the minimum detectable true effect of an intervention—for guidance specific to community college evaluations, see Somers et al. (2023).

Notably, researchers should be mindful of the outcome(s) of interest and the timing of measuring those outcomes—this has implications for what sized effects might realistically be achieved. For example, the size of the effect an intervention might reasonably achieve on marginal credits earned in semester 2 is quite different than the size of the effect an intervention might achieve on cumulative credits earned through semester 4. Thus, MDTE calculations need to be specific with respect to outcome and timing.

Interpretation

The findings presented in this paper begin to provide empirical benchmarks to help researchers and policymakers interpret effect estimates from evaluations of CC interventions. Returning to the example of the DPP intervention highlighted in the introduction, recall that the estimated effect of DPP was 1.8 credits earned after two semesters (Ratledge et al., 2021, Appendix Table B3). This results in an adjusted empirical Bayes impact estimate of 1.6 credits earned, which is about 0.70 standard deviations above the mean effect, and at the 83^rd percentile in the distribution of effects after two semesters.18 Thus, the effect of the DPP intervention is quite large compared to effects of other evaluated interventions.

As noted earlier, the interventions included in our analysis vary in terms of their components and duration (see Table 1). For this reason, in theory, it may be appropriate for researchers to compare the effect of their study to the findings from prior studies of similar interventions. To this end, study-level estimates of intervention effects (OLS, empirical Bayes, and adjusted empirical Bayes) for the 39 interventions are available in a public-use dataset created for this paper (https://www.mdrc.org/the-rct-empirical-benchmarks). The dataset includes estimated effects for each of the 39 studies in the analysis, by semester, for all outcomes (including additional outcomes like credits attempted and credits earned, for total, college-level, and developmental credits), allowing researchers to look at estimated effects for narrower categories of interventions as well as additional outcomes. As more CC studies are conducted, it may be possible to look at effect sizes for different populations, and a broader array of interventions.

Researchers of CC studies can also consider using other types of benchmarks to interpret their findings. One of the limitations of using effects from prior studies as benchmarks is that while they inform what is realistically attainable, they do not necessarily inform what effects are practically meaningful to decision-makers. In K-12 research, several other approaches have been offered for interpreting the practical meaningfulness of effect sizes. This includes comparing a study’s effects to “normative expectations for change or growth” (e.g., comparing a study’s effect to the typical growth made by a student during the year) and policy-relevant performance gaps (e.g., comparing a study’s effect to the outcomes gap between students from families with low versus higher income; Baird & Pane, 2019; Bloom et al., 2008; Hill et al., 2008; Konstantopoulos & Hedges, 2008; Kraft, 2020; Lipsey et al., 2012; Wolf & Harbatkin, 2022).

These approaches could also be used to interpret the practical meaningfulness of findings from CC studies. With respect to normative expectations for growth, the magnitude of a program’s effect on college-level credit accumulation could be characterized relative to normal academic progress towards a degree over various time periods. For example, for students who first enrolled in a public CC in 2012, average credit accumulation over one year nationally was 13.2 credits.19 Therefore, if an intervention’s estimated effect on credit accumulation through two semesters is 3.0 credits, then the intervention could be said to increase credit accumulation by 25% (3.0/12.0) of the national average credit accumulation.

Similarly, comparing estimated effects to inequality in academic outcomes across relevant populations could also be used to characterize impacts. For example, nationally, among students who first enrolled in a public CC in 2017, there is racial inequality in three-year graduation rates of 10.5 percentage points between Hispanic men and White men (U.S. Department of Education, 2021). Thus, an intervention targeting Latino males with a 1.4 percentage point effect on three-year graduation rates could be characterized as reducing racial inequality in graduation rates among Hispanic men and White men by about 13%.

When using normative benchmarks (whether progress towards a degree or racial inequality in academic outcomes), an important consideration is what the reference population should be, which in turn depends on how the findings will be used. The previous examples were based on a national reference population, which may be relevant if the goal is to understand the potential of an intervention to address racial inequality if it were scaled to additional CCs across the country. Information about postsecondary outcomes and achievement gaps for nationally representative samples are readily available from public data sources like the Integrated Postsecondary Education Data System (IPEDS) and the Beginning Postsecondary Students (BPS) study conducted by the National Center for Education Statistics (NCES).20 However, for policymakers or practitioners working at the state or institutional level, benchmarks based on a local normative population could be more appropriate. For example, if an intervention is “home grown” by a state, then state policymakers may find it most useful to interpret the effect sizes from a study relative to the racial inequality in academic outcomes in their state. Similarly, at the colleges that are implementing the intervention being evaluated, administrators may want to know by how much the intervention reduces racial inequality in student outcomes for students at their institution, or even more specifically, for the subset of students who were eligible to receive the intervention. Information on a college’s racial inequality in academic outcomes can often be obtained directly from the institution, and gaps for participating students can be estimated using the study’s control group.

For policymakers and practitioners, practical considerations related to an intervention’s implementation can be as important as its effectiveness. Hence, other powerful approaches for contextualizing a study’s effects are to discuss the intervention’s cost-effectiveness (impact per dollar spent) and its scalability to different contexts (Kraft, 2020). Notably, 19 of the interventions in THE-RCT are part of MDRC’s Intervention Return on Investment (ROI) Tool for Community Colleges (see https://www.mdrc.org/intervention-roi-tool), and thus estimates of the direct costs of these interventions are publicly available. Careful thought would be required to pull this information together to create cost-effectiveness benchmarks—an important area for future investigation that we hope to pursue.

Other Remarks

Comparing a Current Impact Estimate to an Estimated Distribution of Past True Impacts

This appendix describes two scenarios for how to compare a current intervention impact estimate to an estimated distribution of past related true impacts. The first scenario involves a researcher who wants to compare a current impact estimate for a postsecondary intervention to an estimated distribution of corresponding true impacts for postsecondary interventions in Appendix Table A4 of this paper. The second scenario involves a researcher who wants to compare a current impact estimate for any type of intervention to a corresponding distribution of true impacts for related interventions that the researcher will estimate from past and current research findings. In both scenarios, the goal is to compare a current intervention impact to what is known about the distribution of true impacts for related interventions.