Awe CheckFree tool
robots.txt generator
The robots.txt file tells search crawlers which parts of your site to skip. Pick a platform, add your sitemap address and put the result in the site root.
- Free
- No sign-up
- Runs in your browser
Questions and answers
What is robots.txt?
A text file in the site root that tells Google, Bing and other crawlers which addresses to skip. Without it a crawler visits everything it finds.
Does robots.txt remove a page from search?
No. A page blocked in robots.txt can still appear in results through links from other sites, just without a description. To remove it from the index you need a noindex tag, and the page must stay crawlable for the robot to see it.
What should I always block?
Admin, cart, checkout, the account area and internal search: they create many duplicates and do not belong in results. Never block product, article or category pages.
Do I need a Sitemap line?
Yes, one line. The crawler finds the sitemap straight away, without waiting for you to submit it in Search Console.
Can I block the whole site?
Only a staging copy. Blocking the live site removes it from search. It is a common cause of lost traffic after a migration.
What about AI crawlers?
There is a separate generator for ChatGPT, Claude and Perplexity: decide which AI crawlers to allow and add their groups to this file.
Keep reading
Check how search engines see your site
In a minute Awe Check tests robots.txt, the sitemap, indexing, headers and structured data.