Multipoint rank-order study methodology - Observer issues

RATIONALE AND OBJECTIVES. We performed a multipoint rank-order experiment t o evaluate variability in observers' sensitivity to small differences in im age presentation and to assess observers' performance as a function of the type and number of tasks included. METHODS. Five experienced observers were presented with four sets of chest images that had been compressed at five different levels. Each set containe d six images ranging from noncompressed to approximately 60:1-compressed im ages. Observers were asked to review am images of each case side by side an d rank-order the "quality" of each to enable determination of the presence or absence of interstitial disease and/or pneumothoraces. RESULTS. Observers varied significantly in their ability to detect very sma ll differences among the images (P < 0.001), Those who performed well did s o regardless of whether they ranked a specific abnormality in a multidiseas e or a single-disease setting. CONCLUSIONS. Selected observers can reliably detect very small differences among similar images. These readers could be used to confirm or rule out th e need for objective observer performance-type studies.