The effects of subject-defined categories on judgmental accuracy in confidence assessment tasks

The accuracy of confidence judgments can be determined using measures of di scrimination and calibration. The present paper utilizes a new assessment m ethodology that decomposes the confidence assessment task, allowing us to i nvestigate discrimination and calibration skills in greater depth than has been done in previous studies. Researchers investigating the goodness of co nfidence judgments have typically grouped forecasters' assessments into exp erimenter-defined categories, generally in equal widths of .10. In the pres ent research, subjects created their own categories and later assigned conf idence judgments to the categories, separating the tasks of discriminating categories (discrimination) and assigning numbers to categories (calibratio n), Further, the typical assessment procedure assumes that subjects are abl e to discriminate equally across the confidence scale. Since subjects in th e present study defined their own assessment categories, they could locate those categories at any point on the scale. A final issue of interest was w hether subjects were able to determine accurately the number of categories into which they could discriminate, Sixty subjects performed 1 of 2 tasks, general knowledge or forecasting, in both relatively easy and relatively ha rd conditions. Results showed a trade-off in performance: Calibration gener ally became worse as the number of categories increased, while discriminati on generally improved. Overall accuracy was not affected by the number of c ategories used. Further, subjects partitioned categories more at the high e nd of the scale. Finally measures showed that subjects were not accurate in their beliefs about their own discrimination ability. (C) 1999 Academic Pr ess.