Menu
Technical issues2026-06-28
How to check robots.txt: ways to find and test it
Short answer
Check any site's robots.txt by opening yoursite.com/robots.txt in browser — it's always public. To verify syntax and rules, use Google Search Console 'robots.txt Tester' or online validators (technicalseo.com/tools/robots-txt). They show if a URL is allowed and flag syntax errors.
Check any site's robots.txt by opening yoursite.com/robots.txt — it's always public. For deeper checks use Google Search Console or online validators.
Methods
- Browser — open URL directly.
- Google Search Console → robots.txt Tester.
- Online validators: technicalseo.com, ryte.com.
- Screaming Frog with custom robots.txt.
What to check
- No
Disallow: /(closes whole site). - CSS/JS not blocked.
- Sitemap path is correct.
- No typos in paths.
- No BOM at file start.
This question is also asked as:
- •how to check robots.txt
- •robots.txt tester
К
Команда Adaptive Lab
AI-SEO эксперты
Need an expert look at your case?
Free SEO audit within 24 hours — we'll identify your growth bottlenecks and give a concrete action plan.
Related questions
We use cookies for analytics (Google Analytics, Yandex Metrika) and to improve your experience. You can accept or decline analytics cookies. Necessary cookies are always active. Privacy Policy.