AN INTELLIGENT PERSONAL SPIDER (AGENT) FOR DYNAMIC INTERNET INTRANET SEARCHING/

As Internet services based on the World-Wide Web become more popular, information overload has become a pressing research problem. Difficult ies with search on Internet will worsen as the amount of on-line infor mation increases. A scalable approach to Internet search is critical t o the success of Internet services and other current and future Nation al Information Infrastructure (NII) applications. As part of the ongoi ng Illinois Digital Library Initiative project, this research proposes an intelligent personal spider (agent) approach to Internet searching . The approach, which is grounded on automatic textual analysis and ge neral-purpose search algorithms, is expected to be an improvement over the current static and inefficient Internet searches. In this experim ent, we implemented Internet personal spiders based on best first sear ch and genetic algorithm techniques. These personal spiders can dynami cally take a user's selected starting homepages and search for the mos t closely related homepages in the web, based on the links and keyword indexing. A plain, static CGI/HTML-based interface was developed earl ier, followed by a recent enhancement of a graphical, dynamic Java-bas ed interface. Preliminary evaluation results and two working prototype s (available for Web access) are presented. Although the examples and evaluations presented are mainly based on Internet applications, the a pplicability of the proposed techniques to the potentially more reward ing Intranet applications should be obvious. In particular, we believe the proposed agent design can be used to locate organization-wide inf ormation, to gather new, time-critical organizational information, and to support team-building and communication in Intranets. (C) 1998 Els evier Science B.V. All rights reserved.