Menu
Technical issues2026-06-28

What is robots.txt and why do you need it?

Short answer

Robots.txt is a text file in the site root (yoursite.com/robots.txt) that tells search bots which pages they can crawl. It doesn't block indexing directly — only limits crawling. Needed to stop bots wasting crawl budget on tech pages (cart, account, filters) and hide service sections. Mandatory for any site over 50 pages. Always lives strictly in the domain root.

Robots.txt is a text file in the site root that tells search bots which pages they can crawl. Important: it doesn't block indexing directly — only crawling. To block indexing, use noindex meta tag.

Why you need it

  • Save crawl budget — bot doesn't waste time on cart/account.
  • Prevent junk URLs in index.
  • Point to sitemap.xml.
  • Block AI crawlers (GPTBot, ClaudeBot, PerplexityBot since 2024).

Basic structure

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

This question is also asked as:

  • •what is robots.txt
  • •robots.txt purpose
К
Команда Adaptive Lab
AI-SEO эксперты

Need an expert look at your case?

Free SEO audit within 1–2 days — we'll identify your growth bottlenecks and give a concrete action plan.

Get free audit Learn about audit

Related questions

Technical issues

How to create and configure robots.txt: step-by-step

Technical issues

How to check robots.txt: ways to find and test it

Technical issues

What is sitemap.xml and why does a website need it?

Technical issues

What is noindex and how to use it correctly?

All questions