Core robots.txt directives
| Directive | Meaning | Example |
|---|---|---|
| User-agent | Which crawler the following rules are for. * means all crawlers. | User-agent: Googlebot |
| Disallow | A path crawlers must not visit. | Disallow: /wp-admin/ |
| Allow | An exception inside a disallowed path. | Allow: /wp-admin/admin-ajax.php |
| Sitemap | Your sitemap URL, so Google finds your pages fast. | Sitemap: https://example.com/sitemap.xml |
| * and $ | * means “any text”, $ means “end of the URL”. | Disallow: /*.pdf$ |
Common mistakes to avoid
Disallow: / blocks your whole site. That one line removes your site from Google. It’s often forgotten after moving a site from staging.
robots.txt isn’t security. It stops crawling, not access, and anyone can read the file. Private pages need a password.
Blocking doesn’t remove pages. A blocked page can stay in Google if other sites link to it. To remove it, use noindex and allow crawling.
CSS and JS files. Don’t block them; Google needs them to see your page as visitors do.
robots.txt FAQ
Where does robots.txt go?
In your site root, exactly at yoursite.com/robots.txt. In WordPress, SEO plugins like Yoast and Rank Math create it and let you edit it.
How does Google resolve conflicting rules?
It applies the longest (most specific) matching rule, and Allow wins a tie. That’s exactly what this tester does, following RFC 9309.
Should I block AI bots?
Your call. Blocking protects your content from model training but may reduce your presence in answers from tools like ChatGPT. Blocking Google-Extended doesn’t affect Google Search.
Why can’t the tool fetch my file automatically?
Browsers block reading other sites’ files for security. Open yoursite.com/robots.txt, copy its content and paste it into the tester.