AN EVALUATION OF THE RELIABILITY OF PROBABILITY JUDGMENTS ACROSS RESPONSE-MODES AND OVER TIME

Despite the importance of probability assessment methods in behavioral decision theory and decision analysis, little attention has been dire cted at evaluating their reliability and validity. In fact, no compreh ensive study of reliability has been undertaken. Since reliability is a necessary condition for validity, this oversight is significant. The present study was motivated by that oversight. We investigated the re liability of probability measures derived from three response modes: n umerical probabilities, pie diagrams. and odds. Unlike previous studie s, the experiment was designed to distinguish systematic deviations in probability judgments, such as those due to experience or practice, f rom random deviations. It was found that subjects assessed probabiliti es reliably for all three assessment methods regardless of the reliabi lity measures employed. However, a small but statistically significant decrease over time in the magnitudes of assessed probabilities was ob served. This effect was linked to a decrease in subjects' overconfiden ce during the course of the experiment.