INTERRATER RELIABILITY ISSUES IN MULTICENTER TRIALS .1. THEORETICAL CONCEPTS AND OPERATIONAL PROCEDURES USED IN DEPARTMENT-OF-VETERANS-AFFAIRS COOPERATIVE STUDY NUMBER-3941

This article describes a standardized method for establishing and main taining desired levels of interrater reliability (IRR) in multicenter trials. The procedure involves six steps: distribution of procedural g uides, distribution of an introduction tape, initial distribution of p atient interviews to rate, training at the study kickoff meeting, ongo ing IRR monitoring, and group training throughout the study. This meth od is being used in a national Veterans Affairs Cooperative Study (CS #394), involving nine sites to examine the treatment effects of vitami n E on tardive dyskinesia. The six-step standardized process allowed f or early detection of areas of concern in assessment administration. W hen comparing intraclass correlation coefficients (ICCs) at different points in the initial training, the Barnes Akathisia Scale and Anchore d Brief Psychiatric Rating Scale reliability improved from 0.68 to 0.7 4 and from 0.54 to 0.87, respectively. After analyzing the ratings col lected prior to the start of CS #394, data were collected to conduct t he first check on Abnormal Involuntary Movement Scale (AIMS) IRR durin g enrollment; the estimated ICC for the AIMS had decreased from 0.87 t o 0.60. Raters were instructed to re-assess the subjects from the firs t videotape on the AIMS and received additional training. The re-ratin g indicated very good reliability, 0.84. IRR was measured once for the Global Assessment of Functioning Scale resulting in an ICC of 0.90. T he companion article (Part II: Edson et al. 1997, page 59 of this issu e) describes the statistical procedures used to measure IRR.