ASSESSING TRANSCRIPTION AGREEMENT - METHODOLOGICAL ASPECTS

In recent years it has become common practice among speech researchers to report transcription agreement coefficients, when describing findi ngs based on phonetic transcription. Although such coefficients are in tended to give an indication of the degree of transcription accuracy, in reality it is not clear to what extent high agreement coefficients do indeed guarantee great transcription accuracy. In this paper it is argued that the most commonly used agreement coefficient, percentage a greement, has three major disadvantages when applied to phonetic trans cription: (a) it is based on the assumption that agreement between tra nscription symbols is all-or-none, (b) it is strongly influenced by ch ance agreement, and (c) it gives no account of the criteria used for t ranscription alignment. After a detailed discussion of these drawbacks , an alternative approach to calculating transcription (dis)agreement is proposed. In this approach experimentally derived feature matrices are used as input to a program that aligns transcription pairs automat ically and at the same time calculates a (dis)agreement measure for ea ch transcription pair, the average distance. It is argued that this me tric is more adequate to express the degree of (dis)similarity between transcriptions than the more usual percentage agreement. The results of an evaluation experiment corroborate this view.