On setting response criteria for calibrated subjective probability estimates

Framing the issue of subjective probability calibration in signal-detection -theory terms. this paper first proves a theorem regarding the placement of well-calibrated response criteria and then develops an algorithm guarantee d to find such criteria, should they exist. Application of this algorithm t o tasks varying in difficulty and number of response categories shows that perfect calibration is easiest to attain under median difficulty levels (d ' approximate to 1.4) and is practically or theoretically impossible to att ain when the task is either very hard (d ' approximate to 0.5) or very easy let (d ' approximate to 10). Implications for calibration research, includ ing the hard-easy effect, are discussed. (C) 2001 Academic Press.