MetaSpider: Meta-searching and categorization on the Web

It has become increasingly difficult to locate relevant information on the Web, even with the help of Web search engines. Two approaches to addressing the low precision and poor presentation of search results of current searc h tools are studied: meta-search and document categorization. Meta-search e ngines improve precision by selecting and integrating search results from g eneric or domain-specific Web search engines or other resources. Document c ategorization promises better organization and presentation of retrieved re sults. This article introduces MetaSpider, a meta-search engine that has re al-time indexing and categorizing functions. We report in this paper the ma jor components of MetaSpider and discuss related technical approaches. Init ial results of a user evaluation study comparing MetaSpider, NorthernLight, and MetaCrawler in terms of clustering performance and of time and effort expended show that MetaSpider performed best in precision rate, but disclos e no statistically significant differences in recall rate and time requirem ents. Our experimental study also reveals that MetaSpider exhibited a highe r level of automation than the other two systems and facilitated efficient searching by providing the user with an organized, comprehensive view of th e retrieved documents.