Webcrawler

Part of speech: noun

Definitions

1. An automated program that systematically browses the internet to index web content for search engines ; a software tool designed to scan websites and gather data for retrieval purposes ; a bot used by search engines to collect and update information from webpages to facilitate efficient searching
2. A computer script that navigates through websites and collects information to create or update a searchable index ; a technology that automatically explores web links to extract relevant content for search databases ; a mechanical process that traverses internet pages to assemble data for search engine use
3. Software that methodically visits websites to harvest and catalog information for search engine listings ; an automated crawler that follows hyperlinks on the internet to accumulate web page data ; a program that scans and indexes online content to enhance search result accuracy and comprehensiveness

Etymology: A webcrawler is a compound of "web" and "crawler," and the second part is the more telling one. "Crawl" in its older sense meant to move slowly along the ground on hands and knees, and it entered English from Old Norse "krafla," meaning to scrabble with hands and feet. The image of a program creeping methodically from page to page, following every link it finds, comes straight from that physical crawling. The term arose in the early days of the World Wide Web, when search engines first needed automated programs to gather pages before indexing them. "Crawler" had already been used for software that traverses data, so attaching "web" simply specified its territory. The name is a plain functional description rather than a coined invention, and it has stayed in use for the software that builds search engine indexes.

Synonyms: spider, bot, internet bot, web spider, crawler