World Wide Web Wanderer
|
Also referred to as just the Wanderer, this was a perl based web crawler that was first deployed in June, 1993 to measure the size of the World Wide Web. The Wanderer was developed at MIT by Matthew Gray. Later in 1993, it was used to generate an index called the "Wandex", providing the first search engine on the web.
External links
- Growth of the Web Report -- http://www.mit.edu/~mkgray/net/
- Early mention of Wanderer on www-talk -- http://ksi.cpsc.ucalgary.ca/archives/WWW-TALK/www-talk-1993q2.messages/706.html