Menu
Technical issues2026-06-28

How to create and configure robots.txt: step-by-step

Short answer

Creating robots.txt = create a plain text file named 'robots.txt' and put it in your site root. Minimum working file: User-agent: *, Disallow: /admin/, Sitemap: https://yoursite.com/sitemap.xml. For CMS (WordPress, Wix, Shopify) there are built-in settings or plugins. After creating — verify in Google Search Console via 'robots.txt Tester'.

Creating robots.txt = create a plain text file and put it in your site root. Minimum file is 4 lines. After any changes — verify, since an error can block the whole site from indexing.

Manual method

  1. Create file named robots.txt (not .doc).
  2. Paste template (see below).
  3. Save as UTF-8.
  4. Upload to site root via FTP.
  5. Verify: open yoursite.com/robots.txt.

Template

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

Block AI bots (since 2024)

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

This question is also asked as:

  • •how to create robots.txt
  • •robots.txt setup guide
К
Команда Adaptive Lab
AI-SEO эксперты

Need an expert look at your case?

Free SEO audit within 1–2 days — we'll identify your growth bottlenecks and give a concrete action plan.

Get free audit Learn about audit

Related questions

Technical issues

What is robots.txt and why do you need it?

Technical issues

How to check robots.txt: ways to find and test it

Technical issues

What is sitemap.xml and why does a website need it?

Technical issues

What is noindex and how to use it correctly?

All questions