The Effect of Examinee Pass/Fail Information on the Level of Passing Score Set by Expert Judges

Abstract

Many important applications of criterion-referenced testing require that passing or cutting scores be established. One class of procedures for setting passing scores involves the judgments of subject-matter experts. This study investigated the effect on passing scores of providing judges with pass/fail data for the various possible cut-score levels. Passing scores produced by individual judges using a modified version of the Angoff (1971) procedure are compared with the same judges’ passing scores as revised after considering pass/fail data. For the seven cases studied, no consistent effect was seen on the level of passing score averaged across judges. However, a consistent reduction in the variability of passing scores among judges was observed. Judges with more extreme Angoff passing scores tended to become more moderate in their judgments after considering pass/fail information.