SEMANTIC AND GENERATIVE MODELS FOR LOSSY TEXT COMPRESSION

The complementary paradigms of text compression and image compression suggest that there may be potential for applying methods developed for one domain to the other. In image coding, lossy techniques yield comp ression factors that are vastly superior to those of the best lossless schemes and we show that this is also the case for text. This paper i nvestigates the resulting trade-off between subjective quality of the transmission and its compression factor. Two different methods are des cribed, which can be combined into an extremely effective technique th at provides far better compression than the present state of the art a nd yet preserves a reasonable degree of perceived match between the or iginal and received text. The major challenge for lossy text compressi on is the quantitative evaluation of the quality of this match.