Crawler

A crawler is a program that automatically searches and “scans” the Internet. Search engines use crawlers to evaluate the information they collect and then build an index. The search results are then generated on the basis of the ratings.

Definition

You may also know the term as web crawler,(search engine) spider or (search) bot.

This is a program that constantly searches the vastness of the Internet for new web pages and content.Every search engine on the Internet works on the basis of a crawler to fill and update its index. In order to be able to index the almost infinite number of pages on the Internet, these programs work automatically. Different searchbots are responsible for different functions. One crawler can parse the texts while another one reads the ALT tags of graphic files.

Not least because Google offers the market-leading search engine in Germany and most other countries, the Googlebot – Google’s crawler – is best known.

How does a crawler work?

In principle, a searchbot follows every page on the Internet, as long as it can be found. However, this is described in very general terms – in practice it is much more complex. The calling of the pages is fixed in a certain sequence and repeats itself constantly. The pages found are then sorted and evaluated by various algorithms according to certain criteria. The search engine operators do not publish which criteria are involved and how they are evaluated, as these are their trade secrets. It is therefore the task of SEOs to find out how the algorithms think and work. Focused crawlers concentrate on topic-relevant websites, for example. The searchbot is connected to the index of the search engine and lists them there accordingly.

In case you haven’t quite figured out the role of a searchbot yet….

The following video shows among other things how crawling works and how the web pages are ranked.

Mit dem Laden des Videos akzeptieren Sie die Datenschutzerklärung von YouTube.
Mehr erfahren

Video laden

YouTube immer entsperren

The crawler in practice

Through the logfiles of the crawler, a webmaster can get info about who exactly is crawling the server. He also has certain options to deny the searchbot access. For example, if you do not want certain information to be retrieved by the crawler, you can add so-called meta tags in the HTML document. This can also be achieved via the Robots.txt file with the marking: “Disallow:/”. You can also record with which frequency (via Google Search Console) or how many pages the Googlebot crawls, so that e.g. the Googlebot is able to find the right pages. the server performance is not affected (read also Crawl Budget).

Unfortunately, a crawler is not only used for the index of the search engines, but also, for example. for collecting e-mail addresses. A scraper, for example, acts based on content rather than meta-information. This serves the purpose of accessing content and copying or reusing it.

Relevance for SEO

One thing is certain! Without crawlers there would also be no SERP. They provide the foundation and are effectively the manager that collects the web pages. As mentioned above, the Google Search Console is an important tool to influence crawlers and also to determine whether certain pages are not considered at all. So it is essential to know how they work and what purpose they serve.

Each searchbot has only a limited amount of time available per page – also called crawl budget. With SEO as well as improved navigation and file size, you as a website owner can make better use of the Googlebot’s crawl budget, for example. At the same time, the budget increases due to numerous incoming links and a highly frequented page.

Essential instruments to control crawlers like the Googlebot are the robots.txt file as well as the XML sitemap stored in the Google Search Console. In the Google Search Console you can also check if the Googlebot can reach and index all important areas of a website.