Cr. Rabinov et al., COMPARING RELIABILITY OF PERCEPTUAL RATINGS OF ROUGHNESS AND ACOUSTICMEASURES OF JITTER, Journal of speech and hearing research, 38(1), 1995, pp. 26-32
Acoustic analysis is often favored over perceptual evaluation of voice
because it is considered objective, and thus reliable. However, recen
t studies suggest this traditional bias is unwarranted. This study exa
mined the relative reliability of human listeners and automatic system
s for measuring perturbation in the evaluation of pathologic voices. T
en experienced listeners rated the roughness of 50 voice samples (rang
ing from normal to severely disordered) on a 75 mm visual analog scale
. Rating reliability within and across listeners was compared to the r
eliability of jitter measures produced by several voice analysis syste
ms (CSpeech, SoundScope, CSL, and an interactive hand-marking system).
Results showed that overall listeners agreed as well or better than '
'objective'' algorithms. Further, listeners disagreed in predictable w
ays, whereas automatic algorithms differed in seemingly random fashion
s. Finally, listener reliability increased with severity of pathology;
objective methods quickly broke down as severity increased. These fin
dings suggest that listeners and analysis packages differ greatly in t
heir measurement characteristics. Acoustic measures may have advantage
s over perceptual measures for discriminating among essentially normal
voices; however, reliability is not a good reason for preferring acou
stic measures of perturbation to perceptual measures.