Noindex vs disallow: keeping pages out of search

People often mix up noindex and robots.txt Disallow. They do different things.

Disallow (robots.txt)

Stops crawlers from visiting a path. The page can still be indexed if other sites link to it, but without its content.

Noindex (meta tag or header)

Tells search engines not to show the page in results. The crawler must be able to visit the page to see the tag, so do not also block it in robots.txt.

<meta name="robots" content="noindex">

Common situations

• Thank-you pages after a form: noindex.
• Internal site search results: noindex, and optionally block the path once they have dropped out.
• Staging or test sites: password protection, so neither visitors nor crawlers can see them.
• Admin and account areas: sign-in required; a Disallow line keeps crawlers away.
• PDFs you do not want in search: an X-Robots-Tag: noindex header, since PDFs have no meta tags.

The trap

If you add noindex and block the same page in robots.txt, the crawler never sees the noindex, and the page can stay in results. Allow crawling until the page drops out.

Check after changes

A site audit lists pages with noindex. Review the list after every template change to make sure important pages were not hidden by mistake.

Which to use

• Keep a page out of search results: noindex, crawling allowed.
• Save crawl budget on endless filter URLs: Disallow.
• Private content: password protection, not either of these.

Check a page with the noindex checker and indexability checker.