Free, keeps your existing rules
Decide which AI bots may read your site, GPTBot to PerplexityBot, without touching the rules you already have. Fetch your current file, flip the toggles, copy or download the result.
| Bot | Now | What it does | Decision |
|---|
Optional. The file works without it. We'll send the text and the steps for your platform, nothing else unless you ask.
robots.txt lives at the root of the domain. How you get it there depends on the platform.
Shopify generates robots.txt for you. To change it, add templates/robots.txt.liquid to your theme (Online Store > Themes > Edit code > Add a new template > robots.txt). The template replaces the generated file, so loop over Shopify's default groups to keep them, then paste the AI block from the preview after the loop:
.svg){kind=small}
{%- for group in robots.default_groups -%}
{{- group.user_agent }}
{%- for rule in group.rules -%}
{{ rule }}
{%- endfor -%}
{%- if group.sitemap != blank -%}
{{ group.sitemap }}
{%- endif -%}
{%- endfor -%}
# paste the block between "# AI crawlers" and "# end AI crawlers" here
Save, then open yourstore.com/robots.txt and compare it with the copy you fetched above. Shopify's own rules (cart, checkout, account) should still be there.
WordPress serves a virtual robots.txt unless a real file exists in the root. Three options: edit it in Yoast SEO (Tools > File editor) or Rank Math (General Settings > Edit robots.txt) by pasting the whole preview; upload the file to the folder that holds wp-config.php, which overrides the virtual one; or on WordPress 6.x add the AI block through the robots_txt filter in a small plugin. Keep the Disallow: /wp-admin/ group WordPress adds; the builder keeps it when you fetched the current file.
Admin > Content > Design > Configuration > your website row > Edit > Search Engine Robots. Paste the preview into "Edit custom instruction of robots.txt file" and save. Flush the page cache. If the field is empty today, Magento was serving its default; the preview already includes the AI block, so add your own Disallow lines for /checkout/ and /customer/ above it if you want them.
Webflow. Site settings > SEO > Indexing > robots.txt. Paste the preview, save, publish. Webflow writes exactly what you paste.
Framer. Site Settings > General > robots.txt (or under SEO on newer plans). Paste and publish. Framer adds its own sitemap line if you leave it out.
Cloudflare has blocked AI crawlers at the edge by default for new domains since July 2025, before robots.txt is read. Check AI Crawl Control > Crawlers first: an edge 403 for OAI-SearchBot there overrides anything in your file. Set the search and user-triggered bots to Allow in that tab, then use "Managed robots.txt" in the same product to append rules, or serve the preview from your origin as usual.
To enforce a block (not just ask), add a WAF custom rule on the user agent for the training bots, or let AI Crawl Control block them.
Put the file in the web root as robots.txt, served as text/plain with status 200. Next.js: app/robots.ts or public/robots.txt. Cloudflare Pages, Netlify, Vercel: the public/ folder. Then open yoursite.com/robots.txt in a private window and compare it with the preview.
Amazonbot caches robots.txt for up to 30 days and OpenAI and Meta say to allow about a day, so don't expect log changes the same afternoon.
The free audit reads your robots.txt, reports access for all 11 AI bots and checks whether the pages they reach make sense without JavaScript. 48 checks, fixes for your platform, no sign-up to see the score.
The 11 the free audit reports on: OAI-SearchBot, ChatGPT-User and GPTBot (OpenAI), ClaudeBot and Claude-SearchBot (Anthropic), PerplexityBot, Google-Extended, Bingbot, Applebot-Extended, Amazonbot and meta-externalagent. Each has its own user agent, so you can allow the ones that put you in answers and block the ones that only collect training data. The full list by company, with sources, is in AI crawlers and robots.txt.
GPTBot collects pages for training future OpenAI models. OpenAI documents it as separate from OAI-SearchBot (the ChatGPT search index) and ChatGPT-User (fetches when a user asks). Blocking GPTBot does not remove you from ChatGPT search. Whether to block it is a licensing decision, not a visibility one.
You leave Bing and Copilot. Copilot answers draw on the Bing index, and Bingbot is also how Microsoft reads your pages for shopping results. Almost every site should keep it allowed; the builder warns you when it is set to block.
Yes. Paste or fetch your current robots.txt and the builder keeps every group that is not about one of the 11 AI bots, including Shopify's and WordPress's defaults, sitemap lines and crawl delays. It moves the AI bots into one marked block at the end so each bot is decided in exactly one place. Running it again on its own output gives the same file.
No. robots.txt is a request, and OpenAI, Perplexity and Meta all say their user-triggered fetchers may not follow it. To enforce a block you need the edge: Cloudflare's AI Crawl Control, a WAF rule on the user agent, or verified IP ranges. The audit's crawler check tells you what your file says; your logs tell you what bots do.
Free, no account. The builder runs in your browser. Fetching your current file reads the public robots.txt and caches it for ten minutes. Nothing else is stored unless you ask for the file by email.