"'Blocked by robots.txt' and 'Indexed, Though Blocked by robots.txt'"
Updated
Two Search Console statuses for pages your robots.txt file tells Google not to visit, and why blocking a page doesn't keep it out of search.
What it means
"Blocked by robots.txt" means your site's robots.txt file told Googlebot not to visit the page, so Google didn't crawl it and it isn't indexed. "Indexed, though blocked by robots.txt" means Google still put the page in its index without visiting it, based on links to it from other pages.
Google's help article lists the first as "URL blocked by robots.txt". Both statuses come from the same thing: a Disallow rule in the plain text file at yoursite.com/robots.txt that covers the page's address.
| Status | Where it appears | Is the page in Google? | What it usually means |
|---|---|---|---|
| Blocked by robots.txt | "Why pages aren't indexed" table | No | Fine if you meant to block it; a problem if a rule is catching pages you want found |
| Indexed, though blocked by robots.txt | "Improve page experience" table, as a warning | Yes, but Google never read it | You tried to hide the page with robots.txt, which doesn't work, or you're blocking a page by accident |
Why it matters
Robots.txt controls crawling, not indexing. That distinction causes most of the confusion here. Google respects robots.txt and won't fetch a blocked page, but if other pages link to it, Google can still list it in results using only the link and its anchor text. Its snippet will be very limited, often with no description at all.
So the two common mistakes are opposite:
- Blocking a page you want in search. One wrong line can block a whole folder or even the entire site. This happens often after a redesign or site move, when a "keep search engines out" rule from the test site comes along to the live one.
- Using robots.txt to hide a page you don't want in search. It won't reliably work, and it actually stops Google from seeing a noindex tag on that page, because Google can't read a page it isn't allowed to visit.
Plenty of blocking is deliberate and harmless: admin areas, shopping cart and checkout pages, internal search results. Google itself says URLs blocked by robots.txt "are probably intentionally blocked", so don't treat every entry here as an error.
How to fix it
- Decide whether each page should be in search. Open Indexing → Pages in Search Console, click the status, and look through the example addresses. Admin, cart and login pages can stay blocked.
- Confirm which rule is blocking it. Inspect one address with the inspection bar at the top of Search Console; under Page indexing, Crawl allowed? shows whether robots.txt blocked it. Then open
yoursite.com/robots.txtin your browser and find theDisallowline that matches. Search Console's robots.txt report (under Settings) shows which robots.txt files Google found, when it last read them, and any errors.
If the page should be in search:
- Remove or narrow the rule in your robots.txt file. On a site builder this is usually in the SEO settings; on WordPress it's often managed by an SEO plugin. A single
Disallow: /blocks everything. - Ask Google to re-read the file. In the robots.txt report, use the menu next to the file and choose Request a recrawl.
- Request indexing for the most important pages, or wait for Google to recrawl them. See how to request indexing.
If the page should not be in search (including anything under "Indexed, though blocked"):
- Remove the robots.txt block for that page. This feels backwards, but Google has to be able to visit the page to see the next step.
- Add a noindex tag to the page, or password-protect it. Google's guidance is that noindex, not robots.txt, is how you keep a page out of search. See Excluded by 'noindex' tag.
- Wait for Google to recrawl it. Once it sees the noindex tag, the page drops out of the index.
Once you've fixed every page under a status, click Validate fix on its details page. Google says validation typically takes up to about two weeks.
ask datanephew
Is robots.txt blocking pages I want on Google?
Ask datanephew which of your pages are blocked by robots.txt. It reads your Search Console indexing data for free, including whether Google was allowed to crawl each page, and flags blocked pages that used to get search clicks or are in your sitemap, which are the ones most likely blocked by mistake.
Can datanephew edit my robots.txt file?
No. datanephew can't change robots.txt or noindex settings on your site. It can tell you exactly which pages are blocked and what to change, and on the Owner plan it can resubmit your sitemap and notify IndexNow-supporting search engines like Bing once you've fixed it.
Why is a page I blocked still showing up in Google?
That's the "Indexed, though blocked by robots.txt" warning. Ask datanephew to inspect the page: it will confirm robots.txt is stopping Google from crawling it, which is also what stops Google from seeing a noindex tag. The fix is to unblock it and add noindex instead.