Awe CheckFree tool
robots.txt generator for AI crawlers
AI search bots bring visitors from ChatGPT, Claude and Perplexity answers. Training bots collect text for models. Choose who gets in, copy the ready robots.txt and put it in the root of your site.
- Free
- No sign-up
- Runs in your browser
Questions and answers
What happens if I block GPTBot?
ChatGPT stops using your pages to train models. It does not affect whether your site shows up in ChatGPT answers with search: a separate bot, OAI-SearchBot, handles search.
Is Google-Extended a separate crawler?
It is a robots.txt token that controls whether your site is used to train Gemini models. It does not affect rankings in regular Google Search.
If I block AI bots, will I lose traffic?
Blocking search and answer bots removes you from AI answers. Blocking training bots costs you no visibility: they only collect text for models.
Is this protection against copying?
No, it is a request. Well-behaved bots honour it, others ignore it. Real protection needs rules on the server or CDN.
Where does the file go?
In the site root: your-site/robots.txt. If you already have one, move the groups from the result into it and avoid duplicate User-agent lines.
How do I check the result?
Run Awe Check: the AI search and training bot items show who your robots.txt lets in.
Check how AI sees your site
Awe Check tests AI crawler access, text without JavaScript, llms.txt and markup in a minute, plus laws and security.