robots.txt Explained
A file that tells search engine crawlers which parts of your site they're allowed to visit.
What it means
robots.txt is a plain text file at the root of your site (yoursite.com/robots.txt) that tells crawlers which pages or sections they're allowed or not allowed to visit. It's a set of instructions crawlers generally respect, not a security measure — it doesn't hide a page, it just asks well-behaved bots to stay out.
Why it matters
A single wrong line in robots.txt can accidentally block search engines from your entire site, which is a surprisingly common and easy-to-miss mistake after a site migration or redesign. It's worth checking any time search traffic drops unexpectedly.
ask datanephew
Could a robots.txt mistake explain my traffic drop?
It's one of the first things worth ruling out. Ask datanephew to check whether your site's important pages are being blocked from crawling.
Does datanephew check my robots.txt file?
Yes — as part of a crawling and indexing check, it can flag if robots.txt is blocking pages you actually want indexed.