robots.txt
File at site root telling search bots which sections may be crawled and which not.
Full definition
robots.txt — a text file at site root (https://example.com/robots.txt) telling search bots which sections can be crawled.
Basic syntax
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /*?sort=
Sitemap: https://example.com/sitemap.xmlTop 3 mistakes
1. `Disallow: /` — blocks the ENTIRE site from indexing. Often left from dev mode. 2. Blocking CSS/JS via Disallow — Google can't render the page properly, downranks it. 3. Conflict with noindex — if a page is blocked in robots.txt, Google can't read the noindex tag and keeps showing it in SERP.
What robots.txt does NOT do
robots.txt is a recommendation, not a ban. The file doesn't prevent indexing of pages with external links pointing to them. For reliable exclusion from search — use `<meta name="robots" content="noindex">` or the `X-Robots-Tag` HTTP header.
Verification
Google Search Console → "URL Inspection" — shows if a URL is blocked by robots.txt.