Check your robots.txt before Google does.
We read your robots.txt, tell you what it actually does to your search traffic, and let you test any URL against it. Worst problem first.
- Just your domain — we find the file
- Your sitemap's pages checked too
- Results gone in 24 hours
What this checks
Your robots.txt is the first file a search engine reads, and the only one that can remove your entire site from search with a single line. Most of what can go wrong is invisible — the file looks fine, and the damage is in what it means rather than what it says.
So the findings are ordered by what they cost you, not by where they sit in the file:
A robots.txt that returns a server error is the worst thing on this list. Google treats a 5xx as though the file said `Disallow: /`, and keeps doing so for up to thirty days after it starts working again. Every page can be serving a perfect 200 while this is happening.
`Disallow: /` on a live site is usually a staging file that was deployed by accident.
Blocked CSS and JavaScript makes your pages look broken to Google while looking fine to you, because it renders them in a real browser.
A missing `Sitemap:` line is the one thing a robots.txt can add rather than take away.
Not having one is fine
If you have no robots.txt at all, that is a complete and valid answer: it means nothing is restricted. We say so plainly rather than colouring it amber, which is why people end up adding empty files to satisfy a checker. Add one when you have something to say.
Test a URL against your rules
Once we have read your file you can type any path and see whether it is allowed — and which line decided it. This is the part Google removed from Search Console at the end of 2023.
It also shows the rule almost everyone gets wrong: *a group naming a crawler replaces the `` group rather than adding to it.** Adding a `User-agent: Googlebot` section with one Disallow in it does not leave your other rules in force for Google. It throws them away.
robots.txt questions
I do not have a robots.txt. Is that a problem?
No. A 404 means no crawler is restricted, which is a perfectly good state for most sites to be in. The usual reason to add one is to point at your sitemap.
Does robots.txt keep a page out of Google?
No, and this catches people out. It asks crawlers not to fetch a page — the page can still be indexed on the strength of links pointing at it, listed with no description. To keep something out of search, let it be crawled and serve a `noindex` robots meta tag instead. `Noindex:` inside robots.txt has done nothing since 2019.
Why does it say Googlebot ignores my Crawl-delay?
Because it does. Bing and Yandex honour it; Google never has. If Googlebot is taking too much of your server, set the crawl rate in Search Console.
Do you check the robots.txt on a subdomain?
Whichever host you enter, yes — and it matters, because every subdomain has its own. The file at blog.example.com is a different file from the one at example.com, and one says nothing about the other.