German Bragin

Robots.txt generator

Create a valid robots.txt file with allow and disallow rules, a sitemap line, and one-click presets for WordPress and for blocking AI training crawlers.

Presets: 

  

  







By German Bragin · Updated  · Free, no signup, runs in your browser




How to create a robots.txt file


Choose a preset or add your own rules. Paths go one per line, and the tool writes the file in the format crawlers expect: one User-agent group per crawler, followed by Allow, Disallow and an optional Sitemap line. Download it and upload it to the root of your domain.


What robots.txt can and cannot do



WordPress robots.txt


A sensible WordPress file blocks /wp-admin/, keeps admin-ajax.php reachable because themes and plugins use it on the front end, and points to your XML sitemap. The WordPress preset sets this up.


Blocking AI crawlers


Tick the crawlers you want to keep out. Blocking Google-Extended opts your content out of Gemini training without affecting Google Search. If you want your pages cited in AI answers, do the opposite and allow the search-oriented bots such as OAI-SearchBot and PerplexityBot.




Frequently asked questions
Where do I put the robots.txt file?
It must live at the root of your domain, for example https://example.com/robots.txt. Crawlers do not look for it in subfolders, and each subdomain needs its own file.
Does robots.txt hide pages from Google?
No. It asks crawlers not to fetch a URL, but a blocked URL can still appear in results if other pages link to it. To keep a page out of the index, use a noindex meta tag or HTTP header and leave the page crawlable so Google can see it.
How do I block AI crawlers?
Add a separate group for each AI user agent, such as GPTBot, ClaudeBot, CCBot, PerplexityBot and Google-Extended, with Disallow: /. Compliant crawlers respect it, but it is a request, not enforcement.
Does Google support Crawl-delay?
No. Googlebot ignores Crawl-delay. Bing and some other crawlers honour it.
Should I block CSS and JavaScript?
No. Google needs to fetch them to render your pages. Blocking them can hurt how your pages are understood and ranked.


More free tools