How public is the web?: Robots, access, and scholarly communication

This paper examines the use of "Robot Exclusion Protocol" to restrict the a ccess of search engine robots to 10 major American university websites belo nging to institutions recently named among "America's Most Wired" universit ies (Gan, 1997). An analysis of web site searching and interviews with web server administrators at these sites shows that the decision to use this pr ocedure is largely technical and is typically made by the web server admini strator. The implications of this decision for openness in scholarly commun ication and for the future of academic, university-based web publishing are discussed.