Web crawlers

internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing (web spidering)

Googlebot is the web crawler software used by Google that collects documents from the web to build a searchable index for the Google Search engine. This name is actually used to refer to two different types of web crawlers: a desktop crawler (to simulate desktop users) and a mobile crawler (to simulate a mobile user).

Scrapy ( ) is a free and open-source web-crawling framework written in Python. Originally designed for web scraping, it can also be used to extract data using APIs or as a general-purpose web crawler. It is currently maintained by Zyte (formerly Scrapinghub), a web-scraping development and services company.

Qwiki was a New York City–based startup automated video production company acquired by Yahoo! on July 2, 2013 for a reported $50 million. Qwiki released an iPhone app that automatically turns the pictures and videos from a user's camera roll into movies to share. The company's initial product, an iPad application that created video summaries of over 3 million search terms, was downloaded more than 3 million times and named by Apple as the best "Search and Reference" application of 2011.

search engine

Bingbot is a web-crawling robot (type of internet bot), deployed by Microsoft October 2010 to supply Bing. It collects documents from the web to build a searchable index for the Bing search engine. It performs the same as Google's Googlebot.