Robots.txt Generator
Build a robots.txt that lets AI search engines in and keeps AI training crawlers out if you choose, with ready presets for Shopify, WordPress and small business sites.
- Free, no signup
- Results in seconds
- Works for Shopify stores
Check my live robots.txt
See which AI crawlers your current file blocks, then load it here to edit.
Start from a preset
Stay visible in ChatGPT, Claude, Perplexity, Google and Bing answers while opting out of model training. Blocks GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider and Amazonbot.
AI crawlers
Your rules
Paths for crawlers to skip, like admin or checkout pages. The AI crawler choices above are added for you.
Group 1
Use * for every crawler, or a name like SemrushBot.
Google ignores Crawl-delay. Bing and some other crawlers use it.
Full URLs, like https://yourstore.com/sitemap.xml.
- Add your sitemap URL so crawlers can find every page. It's usually /sitemap.xml, or /sitemap_index.xml with Yoast.
robots.txt
txtGoes at your site rootUser-agent: *
Disallow:
# AI training crawlers (blocking them doesn't affect AI search)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: CCBot
User-agent: Bytespider
Disallow: /
What this file tells AI crawlers
- All 11 AI search and assistant crawlers can read your site.
- 8 of 8 AI training crawlers blocked.
Publish it
- Save the file as robots.txt.
- Put it at the root of your site so it opens at yourdomain.com/robots.txt. On WordPress, paste it into your SEO plugin's robots.txt editor or upload it with your host's file manager.
- Open yourdomain.com/robots.txt in a browser to confirm the new version is live.
Then check it with the robots.txt report in Google Search Console and confirm AI crawlers get through with the AI Crawler Checker.
One small file decides which AI engines can read you
robots.txt is the first file ChatGPT, Claude, Perplexity and Google's crawlers read on your site. A few lines in it decide whether you can show up in AI answers, whether your pages are used to train AI models, and whether Google can index you at all. Getting it wrong is easy: most AI crawler lists online mix up search bots and training bots, so people block the crawler that would have recommended them.
- Search crawlers and training crawlers need different answers
- OAI-SearchBot, Claude-SearchBot and PerplexityBot decide whether AI search can cite you. GPTBot, ClaudeBot, CCBot and Google-Extended collect training data. This generator treats them separately, so blocking training never quietly blocks search.
- A typo can hide your whole site
- User-agent: * followed by Disallow: / tells every crawler, Google included, to skip every page. We read the file you build with a standards-based parser and warn you before you publish anything that does that.
- Shopify works differently
- Shopify stores can't upload a robots.txt file. The generator writes the robots.txt.liquid template Shopify expects, keeping Shopify's own default rules so checkout, cart and admin pages stay out of search.
- The crawler list changes
- AI companies add and rename crawlers often. Anthropic's old anthropic-ai and Claude-Web tokens are gone, replaced by ClaudeBot, Claude-SearchBot and Claude-User. The list here is the one our AI Crawler Checker tests against.
How to use the Robots.txt Generator
Check your live file (optional)
Enter your domain to load your current robots.txt. You'll see which AI crawlers it blocks today, and you can load it into the editor to change it rather than starting over.
Pick a preset
Allow AI search and block AI training, allow all AI, a small business or WordPress setup, Shopify, or block everything for a staging site. Choose Shopify if your store runs on it.
Adjust crawlers and rules
Set each AI crawler to allow or block, add paths you want crawlers to skip and add your sitemap URL. The summary under the output shows what the file will actually do.
Copy, publish and verify
Copy or download the file and follow the steps shown for your platform. Then check it in the robots.txt report in Google Search Console and run the AI Crawler Checker.
Robots.txt examples for AI crawlers
Stay in AI search, opt out of AI training
The setup most businesses that care about their content want. ChatGPT, Claude, Perplexity, Google and Bing can still find and cite you.
User-agent: * Disallow: # AI training crawlers (blocking them doesn't affect AI search) User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Meta-ExternalAgent User-agent: Amazonbot User-agent: CCBot User-agent: Bytespider Disallow: /
Block only OpenAI's training crawler
GPTBot is blocked. OAI-SearchBot and ChatGPT-User aren't named, so they follow the User-agent: * group and stay allowed.
User-agent: * Disallow: # AI training crawlers (blocking them doesn't affect AI search) User-agent: GPTBot Disallow: / Sitemap: https://yourstore.com/sitemap.xml
Shopify: robots.txt.liquid that blocks AI training
Keeps Shopify's default rules through robots.default_groups and adds the AI training group after them.
{% comment %}
Keeps Shopify's default rules (they're updated by Shopify) and adds yours.
{% endcomment %}
{% for group in robots.default_groups %}
{{- group.user_agent }}
{%- for rule in group.rules -%}
{{ rule }}
{%- endfor -%}
{%- if group.sitemap != blank -%}
{{ group.sitemap }}
{%- endif -%}
{% endfor %}
# AI training crawlers (blocking them doesn't affect AI search)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: CCBot
User-agent: Bytespider
Disallow: /
Which AI bots should robots.txt allow?
Use these exact tokens in your User-agent lines. Keep AI search and user-triggered crawlers allowed if you want to be recommended; decide on training crawlers based on how you feel about your content training AI models.
| robots.txt token | Company | Type | What it powers | If you block it |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | AI search | ChatGPT search results | You can't be found or cited in ChatGPT search results. |
| ChatGPT-User | OpenAI | User-triggered | Pages ChatGPT opens for a user | OpenAI's assistant can't open your pages when someone asks it to. |
| GPTBot | OpenAI | AI training | Training data for OpenAI models | Your pages are left out of future OpenAI model training. AI search is not affected. |
| Claude-SearchBot | Anthropic | AI search | Claude search results | You can't be found or cited in Claude search results. |
| Claude-User | Anthropic | User-triggered | Pages Claude opens for a user | Anthropic's assistant can't open your pages when someone asks it to. |
| ClaudeBot | Anthropic | AI training | Training data for Claude models | Your pages are left out of future Anthropic model training. AI search is not affected. |
| PerplexityBot | Perplexity | AI search | Perplexity answers | You can't be found or cited in Perplexity answers. |
| Perplexity-User | Perplexity | User-triggered | Pages Perplexity opens for a user | Perplexity's assistant can't open your pages when someone asks it to. |
| Googlebot | AI search | Google Search, AI Overviews and AI Mode | You drop out of Google Search, including AI Overviews and AI Mode. | |
| Google-Extended | AI training | Gemini training and grounding (robots.txt token only) | Google won't use your pages to train Gemini or ground its answers. Google Search and AI Overviews are not affected. | |
| Bingbot | Microsoft | AI search | Bing, Microsoft Copilot and partner AI search | You drop out of Bing and the AI answers built on it, such as Microsoft Copilot. |
| Applebot | Apple | AI search | Siri, Spotlight and Apple Intelligence answers | You can't be found or cited in Siri, Spotlight and Apple Intelligence answers. |
| Applebot-Extended | Apple | AI training | Apple model training (robots.txt token only) | Apple won't use your pages to train its AI models. Siri and Spotlight can still show you. |
| DuckAssistBot | DuckDuckGo | AI search | DuckDuckGo AI-assisted answers | You can't be found or cited in DuckDuckGo AI-assisted answers. |
| MistralAI-User | Mistral | User-triggered | Pages Le Chat opens for a user | Mistral's assistant can't open your pages when someone asks it to. |
| Meta-ExternalAgent | Meta | AI training | Training data for Meta AI models | Your pages are left out of future Meta model training. AI search is not affected. |
| Amazonbot | Amazon | AI training | Alexa answers and Amazon AI models | Amazon may not use your pages for Alexa answers or to train its models. |
| CCBot | Common Crawl | AI training | Open dataset used to train many AI models | Your pages are left out of Common Crawl, an open dataset many AI models are trained on. |
| Bytespider | ByteDance | AI training | Training data for ByteDance models | Your pages are left out of future ByteDance model training. AI search is not affected. |
Questions about the Robots.txt Generator
Letting AI in doesn't mean AI recommends you.
RankBull tracks what ChatGPT, Gemini and Perplexity say about your business every week and publishes the content that moves you into the answer.
Shopify stores change robots.txt through robots.txt.liquid. The generator gives you the exact Shopify version.
Free AI visibility report for your site