Optimizing the number of robots for web search engines

Robots are deployed by a Web search engine for collecting information from different Web servers in order to maintain the currency of its data base of Web pages. In this paper, we investigate the number of robots to be used b y a scorch engine so as to maximize the currency of the data base without p utting an unnecessary load on the network. We use a queueing model to repre sent the system. The arrivals to the queueing system are Web pages brought by the robots; service corresponds to the indexing of these pages. The obje ctive is to find the number of robots, and thus the arrival rate of the que ueing system, such that the indexing queue is neither starved nor saturated . For this, we consider a finite-buffer queueing system and define the cost function to be minimized as a weighted sum of the loss probability and the starvation probability. Under the assumption that arrivals form a Poisson process, and that service times are independent and identically distributed random variables with an exponential distribution, or with a more general service function, we obtain explicit/numerical solutions for the optimal nu mber of robots to deploy.