A COMPARISON OF HUMAN AND STATISTICAL LANGUAGE MODEL PERFORMANCE USING MISSING-WORD TESTS

This paper presents results from a series of missing-word tests, in wh ich a small fragment of text is presented to human subjects who are th en asked to suggest a ranked list of completions. The same experiment is repeated with the WA model, an n-gram statistical language model. F rom the completion data two measures are obtained: (i) verbatim predic tability, which indicates the extent to which subjects nominated exact ly the missing word, and (ii) grammatical class predictability, which indicates the extent to which subjects nominated words of the same gra mmatical class as the missing word. The differences in language model performance and human performance are encouragingly small, especially for verbatim predictability. This is especially significant given that the WA model was able, on average, to use at most half the available context. The results highlight human superiority in handling missing c ontent words. Most importantly, the experiments illustrate the detaile d information one can obtain about the performance of a language model through using missing-word tests.