Assessing reproducibility of data obtained with instruments based on continuous measurements

Data obtained with any research tool must be reproducible, a concept referr ed to as reliability. Three techniques are often used to evaluate reliabili ty of tools using continuous data in aging research: intraclass correlation coefficients (ICC), Pearson correlations, and paired t tests. These are of ten construed as equivalent when applied to reliability. This is not correc t, and may lead researchers to select instruments based on statistics that may not reflect actual reliability. The purpose of this paper is to compare the reliability estimates produced by these three techniques and determine the preferable technique. A hypothetical dataset was produced to evaluate the reliability estimates obtained with ICC, Pearson correlations, and pair ed t tests in thr ee different situations. For each situation two sets of 2 0 observations were created to simulate an intrarater or inter-rater paradi gm, based on 20 participants with two observations pei participant. Situati ons were designed to demonstrate good agreement, systematic bias, ol substa ntial random measurement error. In the situation demonstrating good agreeme nt, all three techniques supported the conclusion that the data were reliab le. In the situation demonstrating systematic bias, the ICC and t test sugg ested the data were not reliable, whereas the Pearson correlation suggested high reliability despite the systematic discrepancy. In the situation repr esenting substantial random measurement error where low reliability was exp ected, the ICC and Peal son coefficient accurately illustrated this. The t test suggested the data were reliable. The ICC is the preferred technique t o measure reliability. Although there are some limitations associated with the use of this technique, they can be overcome.