“AI crawlers” are really several different bots. The robots.txt file lets you decide separately whether your site’s content can be used to train models and whether your site should appear in AI search answers.
Some people say “block AI”, others say “be visible in ChatGPT”, and in all that noise it’s easy to miss one thing: these are two separate decisions. Below I explain which bot does what, show a sample file and point out where a block might be hiding outside robots.txt. At the end: how to choose the right settings for your own business.
Table of contents
Which AI crawlers visit a company website and what are they for?
Each OpenAI bot does something different. And you configure each one independently of the others. According to OpenAI’s crawler documentation, four such programs may visit your site:
- GPTBot collects content that may be used to train generative models.
- OAI-SearchBot is responsible for showing sites in ChatGPT search.
- ChatGPT-User visits a site only when a user asks it to. It doesn’t crawl the web on its own.
- OAI-AdsBot checks ad landing pages in ChatGPT. Does it respect robots.txt? The documentation doesn’t say.
There’s a catch with ChatGPT-User. The robots.txt rules may simply not apply to it, because the visit is requested by a person, not by an automated system.
Blocking GPTBot vs visibility in ChatGPT: two separate decisions
Blocking GPTBot doesn’t remove your site from ChatGPT search. That’s OAI-SearchBot’s job. A Disallow for GPTBot is just a signal: “don’t train on my content”. OAI-SearchBot is a different story, because a site that blocks it won’t appear in ChatGPT search answers (at most as a navigational link). This is where a common mistake happens. Someone adds a blanket block for “all AI bots” and cuts off their own search visibility in the process. If you’re planning on preparing your site for AI Overviews or other AI answers, keep these two settings separate.
Google-Extended: what it changes and what it doesn’t touch
Google-Extended is a token in robots.txt. According to Google’s crawler documentation, it decides whether your content can be used to train future Gemini models and for grounding (backing answers with sources) in the Gemini apps and Vertex AI. It doesn’t affect your site’s presence in Google Search. It isn’t a ranking signal either. There’s also no separate user agent. Googlebot fetches the page as usual, and the token only tells Google what it may use the fetched content for. That’s why looking for Google-Extended in your server logs makes no sense. You won’t find a single entry there.
AI crawlers in robots.txt: a sample configuration
Nothing complicated. A separate User-agent block for each token, each with its own Allow or Disallow rule. The file below blocks model training by OpenAI and Google, while letting the ChatGPT search crawler in:
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /Just remember: as Google’s introduction to robots.txt explains, this file is a request to crawlers, not a security measure. It doesn’t hide a page from search results either. After changing the file you need to wait a while, because according to OpenAI’s notes on its crawlers, their systems may need about a day to pick up the new rules. Robots.txt is also sometimes confused with another file, so it’s worth knowing what the llms.txt file does and how it differs from crawler rules.
Is your hosting already blocking AI bots? How to check in the logs
AI bot blocking often sits somewhere other than robots.txt entirely. In hosting filters, in a WAF, in a CDN or in a security plugin. The result? The file politely lets OAI-SearchBot in, and the server refuses it anyway. Here’s how to check:
- Download the server access logs from your hosting panel.
- Search them for GPTBot, OAI-SearchBot and ChatGPT-User.
- Compare the response codes: 200 means the bot got the page, 403 means it was refused.
- If you see refusals, review your WAF, CDN and security plugin rules.
A handy detail: OpenAI bots may add a robots.txt marker to the user agent when they fetch the file itself. That makes it easy to tell those visits apart from page fetches. And one more thing. When choosing hosting for your website, ask about bot filters right away. It will save you hunting for the culprit later.
Which AI crawlers should you allow and which should you block?
This is a business decision, not a technical one. On one side, visibility in AI answers; on the other, reluctance to hand over content for model training. A typical compromise looks like this: OAI-SearchBot allowed, GPTBot and Google-Extended blocked. A site with unique knowledge, for example paid analyses, may need stronger protection. A service business looking for clients? Usually it pays off more to be visible. But no illusions: letting a bot in gives you a chance to appear in answers, it doesn’t guarantee anything.
Not sure how your site is doing today? A company website SEO audit can help, and it also covers robots.txt and crawler access. In practice I recommend one thing: always check AI crawlers in robots.txt together with your server’s responses. Only then can you see what the bots actually get.
Frequently asked questions
Will blocking GPTBot remove my site from ChatGPT?
No. OAI-SearchBot is responsible for showing sites in ChatGPT search. GPTBot only collects content for model training, so blocking it doesn’t affect search results.
Will blocking Google-Extended lower my rankings in Google?
No. According to Google’s documentation, this token doesn’t affect your presence in Search and isn’t a ranking signal. It only decides whether your content can be used to train Gemini models and for grounding.
Does ChatGPT-User respect robots.txt?
OpenAI’s documentation notes that robots.txt rules may not apply to it, because it acts on a user’s request. And in general, robots.txt is not a security measure at all. Content that should be inaccessible is protected at the server level, for example with a login.
