Free tool
AI Visibility Checker
Search is turning into an answer machine. People ask an AI and read the reply, so the rules that decide whether your site feeds those answers now matter as much as ranking once did. This tool writes the file that sets those rules for you.
Start in Simple for three plain choices, or switch to Advanced to allow or block each AI company one by one. Classic search stays on the whole time, so your SEO never changes. New to the shift? Read The click is dying first.
Your site
We fetch your current robots.txt and merge your existing rules in. Optional, you can skip straight to the choices.
Show up in AI answers?
When someone asks ChatGPT, Perplexity, Claude, or Google's AI about your topic, your site can be quoted and linked. Recommended for most businesses that want to be found.
Let AI companies train on your content for free?
Training crawlers copy your pages to build future models. Blocking them is off by default and does not affect whether you appear in AI answers or normal search.
Allow it when a real person asks an AI to read your page?
Someone pastes your link into an AI and says "summarise this". That's a person choosing to visit, through a tool. Most sites leave this on.
What this does, in plain English
Updates the instant you change anything above.
Your robots.txt
Where to paste this. Save it as a plain file named robots.txt at the top level of your site, so it loads at https://yoursite.com/robots.txt. If you scanned a site, we already folded your sitemap and any custom rules into the text above.
On WordPress? Use an SEO plugin (Yoast or Rank Math) and paste this into its robots.txt editor, or drop the file in your site root over SFTP. Hand-coded site? Put robots.txt in your web root or public folder next to your homepage.
Using Cloudflare? Read this.
robots.txt is a polite request, not a lock. Reputable AI bots honor it; some ignore it. For a hard block that bots can't route around, turn on Block AI Bots in your Cloudflare dashboard under Security, Bots.
Time-sensitive: September 15, 2026. On that date Cloudflare changes its defaults. New sites get AI training and agent bots blocked automatically on ad-supported pages. Existing sites keep control only if the owner sets a preference in Security before then, otherwise a default decides for you. Read Cloudflare's own note, then check your dashboard.
The honest fine print
- robots.txt is advisory. Reputable AI bots honor it; some (like Bytespider, and reportedly Perplexity's user-fetcher) ignore it. For enforcement, use your host or CDN.
- Google-Extended and Applebot-Extended are training opt-outs only. Blocking them does not hurt your Google or Apple search ranking.
- Tokens marked community-reported (Bytespider, cohere-ai) aren't vendor-documented. They're included because they show up widely in the wild.
- Cloudflare blocks AI crawlers by default for new sites, so check those settings even after pasting this file.