Google Search Console
“Blocked by robots.txt”: meaning and how to fix it
Depends on the pageWhat it means
Your robots.txt disallows Googlebot from crawling this URL.
Is it a problem?
Fine for areas you don't want crawled. A serious problem if it blocks pages, CSS or JavaScript you need indexed. Note that robots.txt prevents crawling, not indexing.
Common causes
- A Disallow rule broader than intended (e.g. Disallow: /p blocks /pricing)
- A staging robots.txt deployed to production
- Blocking parameters or folders that contain real pages
How to fix it
- Narrow or remove the Disallow rule
- To keep a page out of Google, allow crawling and use noindex instead
- Test the rule before publishing it
How to check the fix
Use the robots.txt report in Search Console settings to confirm Google fetched the new file.
Find the pages worth fixing first
Upload your Search Console export and get the pages and searches with the most clicks at stake. Free, no signup.