How search engines crawl, index and rank your website

Before a page can rank it must be crawled, then indexed, then ranked. Problems at any step stop it from appearing.

Crawling

Crawlers follow links and read sitemaps to discover pages. They obey your robots.txt file. A page with no links pointing to it and missing from the sitemap may never be found. Check yours with the robots.txt checker and sitemap checker.

Indexing

The search engine reads the page, works out what it is about and stores it. Pages marked noindex, duplicates of other pages, or very thin pages may be left out. The indexability checker shows whether a page can be indexed.

Ranking

For each search, indexed pages are ordered by relevance and quality signals: does the page answer the search, is it trustworthy, does it load quickly, do other sites link to it?

A worked example

Say you publish a new page, /purifiers/ro-7-stage. A crawler finds it through a link on /purifiers and through your sitemap. It fetches the page, sees the title "RO 7-Stage Water Purifier 8L", reads the headings and text, and stores it in the index under topics like RO purifiers and water purifiers. When someone searches "7 stage ro purifier", the page becomes a candidate, and its position depends on how well it answers compared with other candidates.

How to tell which step failed

• Not crawled: the page has no internal links and is missing from the sitemap, or robots.txt blocks it.
• Not indexed: it has noindex, it duplicates another page, or it is very thin.
• Indexed but not ranking: it is in the index, but other pages answer the search better or are more trusted.

A site audit checks the first two steps for your whole site; rank tracking shows the third.

What to do

• Link to every important page from at least one other page.
• Keep an XML sitemap up to date and list it in robots.txt.
• Use noindex only on pages you really want hidden.
• Give each page a clear, unique topic.