File:WebCrawlerArchitecture.svg · Wikimedia Commons · See Wikimedia Commons
web crawler
Sign in to saveAlso known as ant, automatic indexer, web spider, web scutter, web crawlers, spider, search engine spider
internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing (web spidering)
Wikidata facts
- Instance of
- software category
- Subclass of
- Internet bot
- Image
- Robots(dot)txt.png
- Has use
- web scraping
Show 2 more facts
- topic's main category
- Category:Web crawlers
- P13411
- sulfur
Sources (2)
via Wikidata · CC0
~32 min read
Encyclopedic overview
Architecture of a Web crawler
A web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering).
Excerpted from Wikipedia’s “web crawler” article, available under the CC BY-SA 4.0 licence.