On Setting Response Criteria for Calibrated Subjective Probability Estimates.

Framing the issue of subjective probability calibration in signal-detection-theory terms, this paper first proves a theorem regarding the placement of well-calibrated response criteria and then develops an algorithm guaranteed to find such criteria, should they exist. Application of this algorithm to tasks varying in difficulty and number of response categories shows that perfect calibration is easiest to attain under median difficulty levels (d' approximately 1.4) and is practically or theoretically impossible to attain when the task is either very hard (d' approximately 0.5) or very easy (d' approximately 10). Implications for calibration research, including the hard-easy effect, are discussed. Copyright 2001 Academic Press.