What is robots.txt and why do you need it?
Robots.txt is a text file in the site root (yoursite.com/robots.txt) that tells search bots which pages they can crawl. It doesn't block indexing directly — only limits crawling. Needed to stop bots wasting crawl budget on tech pages (cart, account, filters) and hide service sections. Mandatory for any site over 50 pages. Always lives strictly in the domain root.
Robots.txt is a text file in the site root that tells search bots which pages they can crawl. Important: it doesn't block indexing directly — only crawling. To block indexing, use noindex meta tag.
Why you need it
- Save crawl budget — bot doesn't waste time on cart/account.
- Prevent junk URLs in index.
- Point to sitemap.xml.
- Block AI crawlers (GPTBot, ClaudeBot, PerplexityBot since 2024).
Basic structure
User-agent: * Disallow: /admin/ Disallow: /cart/ Allow: / Sitemap: https://yoursite.com/sitemap.xml
This question is also asked as:
- •what is robots.txt
- •robots.txt purpose
Need an expert look at your case?
Free SEO audit within 24 hours — we'll identify your growth bottlenecks and give a concrete action plan.
Related questions
We use cookies for analytics (Google Analytics, Yandex Metrika) and to improve your experience. You can accept or decline analytics cookies. Necessary cookies are always active. Privacy Policy.