Query routing for Web search engines: architecture and experiments

General-purpose search engines such as AltaVista and Lycos are notorious fo r returning irrelevant results in response to user queries. Consequently, t housands of specialized, topic-specific search engines (from VacationSpot.c om to KidsHealth.org) have proliferated on the Web. Typically, topic-specif ic engines return far better results for 'on topic' queries as compared wit h standard Web search engines. However, it is difficult for the casual user to identify the appropriate specialized engine for any given search. It is more natural for a user to issue queries at a particular Web site, and hav e these queries automatically routed to the appropriate search engine(s). T his paper describes an automatic query routing system called e-Pilot. Q-Pil ot has an off-line component that creates an approximate model of each spec ialized search engine's topic. On line, Q-Pilot attempts to dynamically rou te each user query to the appropriate specialized search engines. In our ex periments, Q-Pilot was able to identify the appropriate query category 70% of the time. In addition, Q-Pilot picked the best search engine for the que ry, as one of the top three picks out of its repository of 144 engines, abo ut 40% of the time. This paper reports on Q-Pilot's architecture, the query expansion and clustering algorithms it relies on, and the results of our p reliminary experiments. (C) 2000 Published by Elsevier Science B.V. All rig hts reserved.