Free tool

Does your robots.txt block AI shopping agents?

Enter your store. We read your robots.txt the way ChatGPT, Perplexity, Claude, Google and Apple do, and show which AI agents can reach your homepage and product pages.

Which AI bots matter for a store

AI bots come in two kinds. Assistants and AI search (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User, Googlebot, Bingbot, Applebot) read your pages to answer shoppers and recommend products. Block them and your store drops out of those answers.

Training crawlers (GPTBot, Google-Extended, ClaudeBot, Applebot-Extended, CCBot) collect pages to train models. Blocking them is a reasonable choice and doesn't remove you from AI shopping answers.

The mistakes we see most

A second "User-agent: *" group. Crawlers merge every group for *, so a "Disallow: /" meant for one bad bot ends up blocking everyone, Google included. We found this on a well-known Indian fashion store.
Allow-listing only Google and Bing. "User-agent: *" plus "Disallow: /" with a few named search engines quietly shuts out every AI assistant.
Blocking assistants while meaning to block training. GPTBot is training; OAI-SearchBot and ChatGPT-User are what put you in ChatGPT's shopping answers.

A robots.txt that welcomes AI shopping agents

User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account

# Optional: keep training crawlers out
User-agent: GPTBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

Sitemap: https://yourstore.com/sitemap.xml

On Shopify, edit robots.txt through the robots.txt.liquid theme template. Shopify's default already lets AI agents in.