Adaptive retrieval agents: Internalizing local context and scaling up to the Web

This paper discusses a novel distributed adaptive algorithm and representat ion used to construct populations of adaptive Web agents. These InfoSpiders browse networked information environments on-line in search of pages relev ant to the user, by traversing hyperlinks in an autonomous and intelligent fashion. Each agent adapts to the spatial and temporal regularities of its local context thanks to a combination of machine learning techniques inspir ed by ecological models: evolutionary adaptation with local selection, rein forcement learning and selective query expansion by internalization of envi ronmental signals, and optional relevance feedback. We evaluate the feasibi lity and performance of these methods in three domains: a general class of artificial graph environments, a controlled subset of the Web, and (prelimi narly) the full Web. Our results suggest that InfoSpiders could take advant age of the starting points provided by search engines, based on global word statistics, and then use linkage topology to guide their search on-line. W e show how this approach can complement the current state of the art, espec ially with respect to the scalability challenge.