DATA-COMPRESSION IN FULL-TEXT RETRIEVAL-SYSTEMS

When data compression is applied to full-text retrieval systems, intri cate relationships emerge between the amount of compression, access sp eed, and computing resources required. We propose compression methods, and explore corresponding tradeoffs, for all components of static ful l-text systems such as text databases on CD-ROM. These components incl ude lexical indexes, inverted files, bitmaps, signature files, and the main text itself. Results are reported on the application of the meth ods to several substantial full-text databases, and show that a large, unindexed text can be stored, along with indexes that facilitate fast searching, in less than half its original size-at some appreciable co st in primary memory requirements.