Comparing telephone and mail responses to the CAHPS (TM) survey instrument

OBJECTIVES. The Consumer Assessment of Health Plans (CAHPS(TM)) survey is d esigned to collect member experiences with getting medical care. The object ive was to evaluate the comparability of answers to CAMPS(TM) questions whe n data are collected by mail and by telephone interview. METHODS. TWO studies comparing phone and mail responses used a pretest inst rument with parallel samples drawn from Medicaid beneficiaries in Californi a (n = 217 telephone, 97 mail) and adults with chronic conditions who had h ealth insurance through the State of Washington (n = 98 telephone, 109 mail ). A third study used a revised instrument with two parallel cross-section samples of adults covered through the State of Washington (n = 446 telephon e, 609 mail). Questions covered respondents' experiences with getting medic al care through their health plans. RESULTS. In the first two tests, numerous significant differences were foun d in the rates at which questions that potentially did not apply to all res pondents were answered: some ratings were more positive on the telephone. I n the test of a revised instrument, nine of 58 comparisons differed signifi cantly by mode. The systematic differences in response to questions that di d not apply to all respondents were greatly reduced. Only one of four ratin gs and one of seven multi-item composite measures of quality of care were s ignificantly different by mode. CONCLUSION. Although further steps to reduce the remaining mode effects are needed, the data indicate that when the revised CAHPS(TM) questions are us ed, mode of data collection will have little effect on the key results.