IMPROVING PARTICIPATION AND INTERRATER AGREEMENT IN SCORING AMBULATORY PEDIATRIC ASSOCIATION ABSTRACTS - HOW WELL HAVE WE SUCCEEDED

Objective: To determine whether increasing the number and types of rat ers affected interrater agreement in scoring abstracts submitted to th e Ambulatory Pediatric Association. Methods: In 1990, all abstracts we re rated by each of the 11 members of the board of directors of the Am bulatory Pediatric Association. In 1995, abstracts were reviewed by fo ur to five raters, including eight members of the board of directors, two chairpersons of special interest groups, and 10 regional chairpers ons, for a total of 20 potential reviewers. Submissions were divided i nto the following three categories for review: emergency medicine, beh avioral pediatrics, and general pediatrics. Weighted percentage agreem ent and weighted K scores were computed for 1990 and 1995 abstract sco res. Results: Between 1990 and 1995, the number of abstracts submitted to the Ambulatory Pediatric Association increased from 246 to 407, th e number of reviewers increased from 11 to 20, the weighted percentage agreement between raters remained approximately 79%, and weighted kap pa scores remained less than 0.25. Agreement was not significantly bet ter for the emergency medicine and behavioral pediatrics abstracts tha n for general pediatrics, nor was it better for the raters who reviewe d fewer abstracts than those who reviewed many. Conclusions: The numbe r and expertise of those rating abstracts increased from 1990 to 1995. However, interrater agreement did not change and remained low. Furthe r efforts are needed to improve the interrater agreement.