Interobserver agreement among sleep scorers from different centers in a large dataset

Study Objectives: To evaluate epoch by epoch agreement in sleep stage assig nment between scorers from different laboratories. Design: N/A Methods: 62 NPSGs were selected for analysis from 3 sleep centers (38 diagn ostic studies for sleep disordered breathing [SDB], 10 studies during CPAP titration, and 14 studies in subjects with no sleep related complaints or s leep pathology). The sleep recording montage consisted of at least 2 EEG le ads, left and right EOG and a submental EMG. Scoring was performed manually by 5 experienced sleep technologists. No scorer had knowledge of any other scorers' results. Agreement was tabulated both for sleep stage distributio n and on an epoch by epoch basis for the entire data set and the normal and SDB subsets. Measurements and Results: The mean epoch by epoch agreement between scorers for all records was 73% (range 67-82%). Agreements were higher in the norm al subset (mean 76%, range 65-85%) than in the SDB subset (mean 71%, range 65-78%). There was significant variability in agreement between records and between pairs of scorers. Overall, 75% of epochs had at least 4 of the 5 s corers in agreement on the sleep stage and 96% of epochs had agreement of a t least 3 of the 5 scorers. Conclusions: The lever of agreement in sleep stage assignment varies betwee n scorers, by diagnosis, and by record. The level of agreement between labo ratories is lower than what can be maintained between scorers within the sa me laboratory. This warrants caution when comparing data scored in separate laboratories. The lower agreement in SDB patients supports the generally h eld view that sleep fragmentation makes application of the R&K rules less r eliable.