16 AI bots covered
GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider and more — each with its own allow or block rule.
Advertisement Space (Leaderboard)
Free robots.txt generator — decide which AI bots may crawl your site
Robots.txt Generator for AI Bots
Create robots.txt rules for GPTBot, ClaudeBot, Google-Extended and more. Allow AI search bots, block AI training crawlers, preview live, then copy or download — everything runs in your browser.
16 AI bot rules
Live preview
100% client-side
Generator
Build your robots.txt
Quick presets
AI assistants & answer engines
Bots that fetch pages to cite or answer users
AI training crawlers
Bots that collect content to train models
Advertisement Space (Leaderboard)
Features
Why use this robots.txt generator
Smart presets
One click to allow every AI bot, block training crawlers only, or shut out all of them at once.
Per-bot control
Switch any crawler between allow, block and default with a single click — no syntax to remember.
Extra path rules
Add your own disallowed paths that apply to every user agent, from admin panels to checkout pages.
Sitemap declaration
Optionally append your sitemap.xml URL so compliant crawlers can discover every indexable page.
Privacy by design
Everything is generated in your browser with JavaScript — your domain never leaves your device.
Step by step
How to create a robots.txt file in three steps
- 1
Enter your domain
Type your website URL so the generator can reference it in the sitemap line.
- 2
Choose bot rules
Pick a preset or fine-tune each of the 16 AI crawlers with allow, block or default.
- 3
Publish to your domain root
Download the file, upload it to your site root and verify it loads at /robots.txt.
FAQ
Frequently asked questions
What is a robots.txt file?
A robots.txt file is a plain text file at your domain root that tells crawlers which parts of your site they may access. AI bots and search engines read it before fetching your pages.
How do I block AI bots from training on my content?
Add Disallow: / rules for AI training crawlers such as GPTBot, CCBot, Google-Extended, Bytespider and Applebot-Extended. This generator writes those rules for you in one click.
Does blocking AI training bots affect my Google rankings?
No. Search crawlers such as Googlebot and Bingbot are separate from AI training crawlers. Blocking GPTBot or CCBot does not hurt your search rankings.
Can I allow AI search but block AI training?
Yes. Bots like OAI-SearchBot and PerplexityBot power AI answer engines that cite your pages, while GPTBot and CCBot collect training data. You can set different rules for each group.
Where do I upload the robots.txt file?
Upload it to the root of your domain so it resolves directly at /robots.txt, next to your sitemap.xml and llms.txt files.