DELPHI, THE ISSUE OF RELIABILITY - A QUALITATIVE DELPHI STUDY IN PRIMARY HEALTH-CARE IN THE NETHERLANDS

Problems arise when reliability and accuracy of the Delphi technique a re assessed. So far, reliability has been evaluated by comparing two o r more Delphi studies on the same subject, using Pearson, kappa, or ra nk-order coefficients. However, this strategy is not very effective or informative. When numerical ratings of repondents are available and t hese ratings are normally distributed, reliability of Delphi can be ev aluated in a more accurate and effective way by means of the intraclas s correlation coefficient. If the investigator expects any systematic effects of factors characterizing the design of a study, these effects can be extracted from the error variance and analyzed separately. A s econd advantage of the intraclass correlation coefficient is that it c an be derived from the ratings of only one group of respondents. Colle cting extra data is not necessary. By using this strategy, the evaluat ion of reliability becomes feasible in individual applications. The co ncept is systematically elaborated in a Delphi study carried out to de velop evaluation criteria for the performance of family physicians con sulted by patients having nonspecific abdominal complaints.