robots.txt generator, with AI crawler control.
Generate a valid robots.txt in seconds, and decide exactly which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) can use your content. No sign-up.
Click to BLOCK an AI crawler from using your content (leave off to allow):
// Save as robots.txt at your domain root: https://yoursite.com/robots.txt
robots.txt is now an AI-content decision.
robots.txt has always told search crawlers where they can go. Now it also decides whether AI companies can use your content to train models and power answers. Crawlers like GPTBot, ClaudeBot and PerplexityBot each respect their own directives, so you can allow the ones that cite you and block the ones that only take.
Most brands should allow the AI search crawlers (being usable is how you get cited) while making a deliberate choice about training bots. If you are unsure what the right posture is for your business, that is a conversation worth having, it is part of a modern technical SEO setup.
Get your whole technical setup right.
robots.txt is one file. A free technical audit checks everything the engines see.