User-agent: * Allow: / Disallow: /admin/ Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /signin Disallow: /signup Disallow: /search/suggestions # ===================================================================== # AI crawlers — explicit per-bot rules rather than relying on the # wildcard above, since some AI crawlers behave inconsistently with # generic User-agent: * fallback. All currently set to Allow, since for # an e-commerce brand, blocking AI crawlers generally costs more in lost # visibility (product mentions/citations in ChatGPT, Perplexity, Gemini, # etc.) than it protects. Flip any single bot to "Disallow: /" if you'd # rather opt out of it specifically — see the category note on each. # ===================================================================== # --- OpenAI --- # GPTBot: model training. OAI-SearchBot: powers ChatGPT's live search/citations. # ChatGPT-User: fetched on-demand when a user asks ChatGPT to read a page. User-agent: GPTBot Allow: / Disallow: /admin/ Disallow: /cart Disallow: /checkout Disallow: /account User-agent: OAI-SearchBot Allow: / Disallow: /admin/ Disallow: /cart Disallow: /checkout Disallow: /account User-agent: ChatGPT-User Allow: / Disallow: /admin/ Disallow: /cart Disallow: /checkout Disallow: /account # --- Anthropic --- # ClaudeBot: model training. Claude-SearchBot: search indexing. # Claude-User: fetched on-demand when a user asks Claude to read a page. User-agent: ClaudeBot Allow: / Disallow: /admin/ Disallow: /cart Disallow: /checkout Disallow: /account User-agent: Claude-SearchBot Allow: / Disallow: /admin/ Disallow: /cart Disallow: /checkout Disallow: /account User-agent: Claude-User Allow: / Disallow: /admin/ Disallow: /cart Disallow: /checkout Disallow: /account User-agent: anthropic-ai Allow: / Disallow: /admin/ Disallow: /cart Disallow: /checkout Disallow: /account # --- Perplexity --- # PerplexityBot: search index. Perplexity-User: on-demand fetch. # Note: Perplexity has been documented running stealth crawlers that # don't identify themselves or honor robots.txt — this rule only # governs the bots that do. User-agent: PerplexityBot Allow: / Disallow: /admin/ Disallow: /cart Disallow: /checkout Disallow: /account User-agent: Perplexity-User Allow: / Disallow: /admin/ Disallow: /cart Disallow: /checkout Disallow: /account # --- Google --- # Googlebot (regular Search + AI Overviews/AI Mode) is already covered by # the wildcard rule above — you cannot separate AI Overviews from Search # itself, both run through Googlebot. Google-Extended is the separate # token that only controls Gemini/Vertex AI *training* — blocking it # does not affect your normal Search ranking. User-agent: Google-Extended Allow: / Disallow: /admin/ Disallow: /cart Disallow: /checkout Disallow: /account # --- Meta --- User-agent: Meta-ExternalAgent Allow: / Disallow: /admin/ Disallow: /cart Disallow: /checkout Disallow: /account # --- Apple --- User-agent: Applebot-Extended Allow: / Disallow: /admin/ Disallow: /cart Disallow: /checkout Disallow: /account # --- Amazon (shopping-assistant relevant for an e-commerce catalog) --- User-agent: Amazonbot Allow: / Disallow: /admin/ Disallow: /cart Disallow: /checkout Disallow: /account # --- ByteDance (has a documented history of ignoring robots.txt — # this rule is honest documentation of intent, not a guarantee) --- User-agent: Bytespider Allow: / Disallow: /admin/ Disallow: /cart Disallow: /checkout Disallow: /account # --- Common Crawl (widely reused by many AI labs for training) --- User-agent: CCBot Allow: / Disallow: /admin/ Disallow: /cart Disallow: /checkout Disallow: /account # --- Cohere --- User-agent: cohere-ai Allow: / Disallow: /admin/ Disallow: /cart Disallow: /checkout Disallow: /account Sitemap: https://www.seedmix.in/sitemap.xml