World Wide Web Wanderer Guide: History, Origin, and More

What Is World Wide Web Wanderer?: Explained

The brilliant idea of the World Wide Web was devised in the spring of 1989 in the head of Tim Berners-Lee, a physicist at CERN, but it didn’t gain any widespread popular use until the remarkable NCSA Mosaic web browser was introduced at the beginning of 1993.

In the spring of 1993, just months after the release of Mosaic, Matthew Gray studied physics at the Massachusetts Institute of Technology (MIT) and was one of the three members of the Student Information Processing Board (SIPB) who set up the site www.mit.edu. He decided to write a program, called the World Wide Web Wanderer, to systematically traverse the web and collect sites.

The Wanderer was first functional in the spring of 1993 and became the first automated web agent (spider or web crawler). It certainly did not reach every site on the web, but it was run with a consistent methodology, hopefully yielding consistent data for the growth of the web.

Wanderer was written using the Perl language. While crawling the web, it generated an index called Wandex — the first web database. Initially, the Wanderer counted only web servers but shortly after its introduction, it started to capture URLs as it went along.

Matthew Gray’s Wanderer created quite a controversy at the time, partially because early versions of the program ran rampant through the web and caused a noticeable network performance degradation. This degradation occurred because it would access the same page hundreds of times a day. The Wanderer soon amended its ways, but the controversy over whether spiders were good or bad for the Internet remained for some time.

Wanderer certainly was not the Internet’s first search engine, it was the Archie of Alan Emtage, but Wanderer was the first web robot, and, with its index Wandex, clearly had the potential to become the first general-purpose web search engine, years before Yahoo, Google, or Yandex. Mathew Gray however does not make this claim and he always stated that this was not its purpose. Wanderer inspired a number of programmers to follow up on the idea of web robots.

The Difference Between World Wide Web Wanderer vs. WebCrawler

WebCrawler, created by Brian Pinkerton in 1994, is considered the first true web crawler-powered search engine. While some may claim that the title for Wandex is due to its potential, it was never designed to be used in this way. Wandex lacked some critical features to make it a general-purpose search engine.

One of the major innovations of WebCrawler was its full-text searchability. This ability made it popular and highly functional. It continues to operate as a search engine, although not as popular as Google, Yahoo, Bing, Yandex, or Baidu.

World Wide Web Wanderer Release History

Matthew was initially motivated primarily to discover new sites, as the web was still a relatively small place (as of the World Wide Web Wanderer start date in 1993 the total number of websites all over the world was about 100, and in June of 1995, even with the phenomenal growth of the Internet, the number of web servers increased to a point where one in every 270 machines on the Internet is a web server). As the web started to grow rapidly after 1993, the focus quickly changed to charting its entire growth.

World Wide Web Wanderer: End of Development

Despite the many uses of the World Wide Web Wanderer as a web crawler, it was discontinued in 1995. Matthew Gray continued to work in the industry and many other web crawlers and bots have been designed since the original start date of the World Wide Web Wanderer.