Deep web
|
|
The deep web (or invisible web or hidden web) is the name given to pages on the World Wide Web that are not indexed by search engines. It consists of pages which are not linked to by other pages, such as Dynamic Web pages based on responses to database queries. The Deep Web also includes sites that require registration or otherwise limit access to their pages. It should not be confused with the term dark web or Dark internet which refers to machines or network segments not connected to the Internet. While deep web content is visible to people but not to search engines, dark internet content is accessible by neither people nor search engines.
Search engines use web crawlers that follow hyperlinks. Such crawlers typically do not submit queries to databases due to the potential infinitude of queries that can be made to a single database. It has been noted that this can be (partially) overcome by having links to query results, thus increasing Google-style PageRank results for a theoretical member of the deep web.
In a 2000 study by the search company BrightPlanet, the inaccessible part of the web was estimated to be about 500 times larger, in terms of number of documents, than what search engines already provide access to. Any such figures must be taken with caution, however, due to the difficulty of distinguishing between genuinely different documents and documents that merely represent different database views of the same content.
In 2005, Yahoo! made a small part of the deep web searchable by releasing Yahoo! Subscriptions. This search engine searches through a few subscription-only web sites.
References
- Gary Price & Chris Sherman. The Invisible Web : Uncovering Information Sources Search Engines Can't See. CyberAge Books, July 2001. ISBN 091096551X
External links
- http://library.albany.edu/internet/deepweb.html
- http://library.rider.edu/scholarly/rlackie/Invisible/Inv_Web_Main.html
- http://www.invisible-web.net/
- BrightPlanet - Deep Web White Paper (http://www.brightplanet.com/technology/deepweb.asp)
- Ez2Find :: Invisible Web (http://ez2find.com/channel/advanced_search.php) Search the deep web with specialized search engines.
Related research
- QProber: Classifying and Searching "Hidden-Web" Databases (http://qprober.cs.columbia.edu/)
- MetaQuerier: Exploring and Integrating the Deep Web (http://metaquerier.cs.uiuc.edu/)
- Yahoo! Subscriptions (http://search.yahoo.com/subscriptions)de:Deep Web