Menu
Technical issues2026-06-28

How to check robots.txt: ways to find and test it

Short answer

Check any site's robots.txt by opening yoursite.com/robots.txt in browser — it's always public. To verify syntax and rules, use Google Search Console 'robots.txt Tester' or online validators (technicalseo.com/tools/robots-txt). They show if a URL is allowed and flag syntax errors.

Check any site's robots.txt by opening yoursite.com/robots.txt — it's always public. For deeper checks use Google Search Console or online validators.

Methods

  1. Browser — open URL directly.
  2. Google Search Console → robots.txt Tester.
  3. Online validators: technicalseo.com, ryte.com.
  4. Screaming Frog with custom robots.txt.

What to check

  • No Disallow: / (closes whole site).
  • CSS/JS not blocked.
  • Sitemap path is correct.
  • No typos in paths.
  • No BOM at file start.

This question is also asked as:

  • •how to check robots.txt
  • •robots.txt tester
К
Команда Adaptive Lab
AI-SEO эксперты

Need an expert look at your case?

Free SEO audit within 1–2 days — we'll identify your growth bottlenecks and give a concrete action plan.

Get free audit Learn about audit

Related questions

Technical issues

What is robots.txt and why do you need it?

Technical issues

How to create and configure robots.txt: step-by-step

Technical issues

How to check and find sitemap.xml of a website?

Technical issues

How to check if a page or article is indexed in Google and Bing?

All questions