Mon–Sat, 09:00–18:00 PKTMarketing, websites and courses · Mon–Sat, 09:00–18:00 PKT
SEO
robots.txt is a text file at a site's root that tells crawlers which paths they may or may not request. It controls crawling, not indexing.
It uses simple rules: a user-agent line naming a crawler, followed by Allow and Disallow paths. It is also where you point crawlers to your sitemap.
Blocking a page in robots.txt does not reliably remove it from search; Google can still index the URL if other sites link to it. Use a noindex tag for that.
Example
A site disallows /api/ so crawlers do not waste time on data endpoints, and allows AI crawlers such as GPTBot so it can appear in AI answers.
Why it matters
One wrong line in robots.txt can block an entire site from search, so it is one of the first files checked in any audit.
Keep learning
SEO
Tell us what you are trying to achieve and we will recommend the right approach, a timeline, and a quote.
Available for new projects — Replies within one business day
