S SEO Suite
Sign in Create account
SEO tools / Links, crawling and indexing

Website crawler

Crawl a website like a search engine: every URL with status, title, depth, links in, words, speed and problems, with a CSV download.

Free up to 500 pages (2,000 when signed in). The crawl runs while this page is open.

Options
0 = no limit.
One per line; * matches anything.

More links, crawling and indexing tools

Empty link text checkerLinks with no text, label or image alt that search engines cannot read. Too many links checkerCount every link on a page and flag pages with more than 300. URL redirect checkerFollow every redirect hop of a URL with its status code. 404 page checkerDoes the site answer missing pages with a real 404? Broken link checkerCheck every link on a page and find the broken ones. Search engine spider simulatorSee a page the way Googlebot or Bingbot reads it: title, description, robots rul... Robots.txt checker and testerRead a robots.txt and test whether a URL is allowed for a bot. Internal links checkerList every internal link on a page.

How to use the website crawler

Crawl a website like a search engine: every URL with status, title, depth, links in, words, speed and problems, with a CSV download. Enter the page address above and press the button. Results appear on this page and you can share the link.

Visitors can run 30 checks a day. Create a free account for projects, site audits, rank tracking and saved keyword lists.