Google Search Console
“Indexed, though blocked by robots.txt”: meaning and how to fix it
Usually needs fixingWhat it means
Google indexed the URL without crawling it, because links point to it, even though robots.txt blocks crawling. It may show in results with no description.
Is it a problem?
Usually yes: the URL is in Google but you can't control how it looks, and a noindex on the page can't be seen.
Common causes
- Using robots.txt to try to keep pages out of Google
- Blocked URLs that have internal or external links
How to fix it
- If the page should be out of Google: allow crawling in robots.txt and add noindex
- If it should be indexed: remove the Disallow rule
How to check the fix
Validate fix after changing robots.txt.
Find the pages worth fixing first
Upload your Search Console export and get the pages and searches with the most clicks at stake. Free, no signup.