How to create and configure robots.txt: step-by-step
Creating robots.txt = create a plain text file named 'robots.txt' and put it in your site root. Minimum working file: User-agent: *, Disallow: /admin/, Sitemap: https://yoursite.com/sitemap.xml. For CMS (WordPress, Wix, Shopify) there are built-in settings or plugins. After creating — verify in Google Search Console via 'robots.txt Tester'.
Creating robots.txt = create a plain text file and put it in your site root. Minimum file is 4 lines. After any changes — verify, since an error can block the whole site from indexing.
Manual method
- Create file named
robots.txt(not .doc). - Paste template (see below).
- Save as UTF-8.
- Upload to site root via FTP.
- Verify: open yoursite.com/robots.txt.
Template
User-agent: * Disallow: /admin/ Disallow: /cart/ Allow: / Sitemap: https://yoursite.com/sitemap.xml
Block AI bots (since 2024)
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: PerplexityBot Disallow: /
This question is also asked as:
- •how to create robots.txt
- •robots.txt setup guide
Need an expert look at your case?
Free SEO audit within 24 hours — we'll identify your growth bottlenecks and give a concrete action plan.
Related questions
We use cookies for analytics (Google Analytics, Yandex Metrika) and to improve your experience. You can accept or decline analytics cookies. Necessary cookies are always active. Privacy Policy.