The effect of panel membership and feedback on ratings in a two-round Delphi survey - Results of a randomized controlled trial

BACKGROUND. Past observational studies of the RAND/UCLA Appropriateness Met hod have shown that the composition of panels affects the ratings that are obtained, Panels of mixed physicians make different judgments from panels o f single specialty physicians, and physicians who use a procedure are more likely to rate it more highly than those who do not. OBJECTIVES. To determine the effect of using physicians and health care man agers within a panel designed to assess quality indicators for primary care and to test the effect of different types of feedback within the panel pro cess. METHOD. A two-round postal Delphi survey of health care managers and family physicians rated 240 potential indicators of quality of primary care in th e United Kingdom to determine their face validity. Following round one, equ al numbers of managers and physicians were randomly allocated to receive ei ther collective (whole sample) or group-only (own professional group only) feedback, thus, creating four subgroups of two single-specialty panels and two mixed panels. RESULTS. Overall, managers rated the indicators significantly higher than p hysicians, Second-round stores were moderated by the type of feedback recei ved with those receiving collective feedback influenced by the other profes sional group. CONCLUSIONS. This paper provides further experimental evidence that consens us panel judgments are influenced both by panel composition and by the type of feedback which is given to participants during the panel process. Caref ul attention must be given to the methods used to conduct consensus panel s tudies, and methods need to be described in detail when such studies are re ported.