{"id":30863,"date":"2026-10-27T09:15:00","date_gmt":"2026-10-27T08:15:00","guid":{"rendered":"https:\/\/www.web-systems.pl\/?p=30863"},"modified":"2026-10-23T08:53:27","modified_gmt":"2026-10-23T07:53:27","slug":"ai-crawlers-gptbot-oai-searchbot-robots-txt","status":"publish","type":"post","link":"https:\/\/www.web-systems.pl\/en\/ai-crawlers-gptbot-oai-searchbot-robots-txt\/","title":{"rendered":"AI Crawlers on a Company Website: GPTBot, OAI-SearchBot and robots.txt"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">&#8220;AI crawlers&#8221; are really several different bots. The robots.txt file lets you decide separately whether your site&#8217;s content can be used to train models and whether your site should appear in AI search answers.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Some people say &#8220;block AI&#8221;, others say &#8220;be visible in ChatGPT&#8221;, and in all that noise it&#8217;s easy to miss one thing: these are two separate decisions. Below I explain which bot does what, show a sample file and point out where a block might be hiding outside robots.txt. At the end: how to choose the right settings for your own business.<\/p>\n\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_86 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of contents<\/p>\n<span class=\"ez-toc-title-toggle\"><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.web-systems.pl\/en\/ai-crawlers-gptbot-oai-searchbot-robots-txt\/#Which_AI_crawlers_visit_a_company_website_and_what_are_they_for\" >Which AI crawlers visit a company website and what are they for?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.web-systems.pl\/en\/ai-crawlers-gptbot-oai-searchbot-robots-txt\/#Blocking_GPTBot_vs_visibility_in_ChatGPT_two_separate_decisions\" >Blocking GPTBot vs visibility in ChatGPT: two separate decisions<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.web-systems.pl\/en\/ai-crawlers-gptbot-oai-searchbot-robots-txt\/#Google-Extended_what_it_changes_and_what_it_doesnt_touch\" >Google-Extended: what it changes and what it doesn&#8217;t touch<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.web-systems.pl\/en\/ai-crawlers-gptbot-oai-searchbot-robots-txt\/#AI_crawlers_in_robotstxt_a_sample_configuration\" >AI crawlers in robots.txt: a sample configuration<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.web-systems.pl\/en\/ai-crawlers-gptbot-oai-searchbot-robots-txt\/#Is_your_hosting_already_blocking_AI_bots_How_to_check_in_the_logs\" >Is your hosting already blocking AI bots? How to check in the logs<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/www.web-systems.pl\/en\/ai-crawlers-gptbot-oai-searchbot-robots-txt\/#Which_AI_crawlers_should_you_allow_and_which_should_you_block\" >Which AI crawlers should you allow and which should you block?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/www.web-systems.pl\/en\/ai-crawlers-gptbot-oai-searchbot-robots-txt\/#Frequently_asked_questions\" >Frequently asked questions<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/www.web-systems.pl\/en\/ai-crawlers-gptbot-oai-searchbot-robots-txt\/#Will_blocking_GPTBot_remove_my_site_from_ChatGPT\" >Will blocking GPTBot remove my site from ChatGPT?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/www.web-systems.pl\/en\/ai-crawlers-gptbot-oai-searchbot-robots-txt\/#Will_blocking_Google-Extended_lower_my_rankings_in_Google\" >Will blocking Google-Extended lower my rankings in Google?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/www.web-systems.pl\/en\/ai-crawlers-gptbot-oai-searchbot-robots-txt\/#Does_ChatGPT-User_respect_robotstxt\" >Does ChatGPT-User respect robots.txt?<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Which_AI_crawlers_visit_a_company_website_and_what_are_they_for\"><\/span>Which AI crawlers visit a company website and what are they for?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Each OpenAI bot does something different. And you configure each one independently of the others. According to <a href=\"https:\/\/developers.openai.com\/api\/docs\/bots\" rel=\"nofollow noopener\" target=\"_blank\">OpenAI&#8217;s crawler documentation<\/a>, four such programs may visit your site:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>GPTBot<\/strong> collects content that may be used to train generative models.<\/li>\n<li><strong>OAI-SearchBot<\/strong> is responsible for showing sites in ChatGPT search.<\/li>\n<li><strong>ChatGPT-User<\/strong> visits a site only when a user asks it to. It doesn&#8217;t crawl the web on its own.<\/li>\n<li><strong>OAI-AdsBot<\/strong> checks ad landing pages in ChatGPT. Does it respect robots.txt? The documentation doesn&#8217;t say.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">There&#8217;s a catch with ChatGPT-User. The robots.txt rules may simply not apply to it, because the visit is requested by a person, not by an automated system.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Blocking_GPTBot_vs_visibility_in_ChatGPT_two_separate_decisions\"><\/span>Blocking GPTBot vs visibility in ChatGPT: two separate decisions<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Blocking GPTBot doesn&#8217;t remove your site from ChatGPT search. That&#8217;s OAI-SearchBot&#8217;s job. A Disallow for GPTBot is just a signal: &#8220;don&#8217;t train on my content&#8221;. OAI-SearchBot is a different story, because a site that blocks it won&#8217;t appear in ChatGPT search answers (at most as a navigational link). This is where a common mistake happens. Someone adds a blanket block for &#8220;all AI bots&#8221; and cuts off their own search visibility in the process. If you&#8217;re planning on <a href=\"https:\/\/www.web-systems.pl\/en\/ai-overviews-prepare-company-website-for-ai-answers\/\">preparing your site for AI Overviews<\/a> or other AI answers, keep these two settings separate.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Google-Extended_what_it_changes_and_what_it_doesnt_touch\"><\/span>Google-Extended: what it changes and what it doesn&#8217;t touch<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Google-Extended is a token in robots.txt. According to <a href=\"https:\/\/developers.google.com\/crawling\/docs\/crawlers-fetchers\/google-common-crawlers\" rel=\"nofollow noopener\" target=\"_blank\">Google&#8217;s crawler documentation<\/a>, it decides whether your content can be used to train future Gemini models and for grounding (backing answers with sources) in the Gemini apps and Vertex AI. It doesn&#8217;t affect your site&#8217;s presence in Google Search. It isn&#8217;t a ranking signal either. There&#8217;s also no separate user agent. Googlebot fetches the page as usual, and the token only tells Google what it may use the fetched content for. That&#8217;s why looking for Google-Extended in your server logs makes no sense. You won&#8217;t find a single entry there.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"AI_crawlers_in_robotstxt_a_sample_configuration\"><\/span>AI crawlers in robots.txt: a sample configuration<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Nothing complicated. A separate User-agent block for each token, each with its own Allow or Disallow rule. The file below blocks model training by OpenAI and Google, while letting the ChatGPT search crawler in:<\/p>\n\n\n\n<pre><code>User-agent: GPTBot\nDisallow: \/\n\nUser-agent: Google-Extended\nDisallow: \/\n\nUser-agent: OAI-SearchBot\nAllow: \/<\/code><\/pre>\n<p class=\"wp-block-paragraph\">Just remember: as <a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/intro\" rel=\"nofollow noopener\" target=\"_blank\">Google&#8217;s introduction to robots.txt<\/a> explains, this file is a request to crawlers, not a security measure. It doesn&#8217;t hide a page from search results either. After changing the file you need to wait a while, because according to OpenAI&#8217;s notes on its crawlers, their systems may need about a day to pick up the new rules. Robots.txt is also sometimes confused with another file, so it&#8217;s worth knowing <a href=\"https:\/\/www.web-systems.pl\/en\/llms-txt-file-what-it-does-for-seo-and-how-to-use-it\/\">what the llms.txt file does<\/a> and how it differs from crawler rules.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Is_your_hosting_already_blocking_AI_bots_How_to_check_in_the_logs\"><\/span>Is your hosting already blocking AI bots? How to check in the logs<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">AI bot blocking often sits somewhere other than robots.txt entirely. In hosting filters, in a WAF, in a CDN or in a security plugin. The result? The file politely lets OAI-SearchBot in, and the server refuses it anyway. Here&#8217;s how to check:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Download the server access logs from your hosting panel.<\/li>\n<li>Search them for GPTBot, OAI-SearchBot and ChatGPT-User.<\/li>\n<li>Compare the response codes: 200 means the bot got the page, 403 means it was refused.<\/li>\n<li>If you see refusals, review your WAF, CDN and security plugin rules.<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">A handy detail: OpenAI bots may add a robots.txt marker to the user agent when they fetch the file itself. That makes it easy to tell those visits apart from page fetches. And one more thing. When choosing <a href=\"https:\/\/www.web-systems.pl\/en\/business-website-hosting-how-to-choose-server\/\">hosting for your website<\/a>, ask about bot filters right away. It will save you hunting for the culprit later.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Which_AI_crawlers_should_you_allow_and_which_should_you_block\"><\/span>Which AI crawlers should you allow and which should you block?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">This is a business decision, not a technical one. On one side, visibility in AI answers; on the other, reluctance to hand over content for model training. A typical compromise looks like this: OAI-SearchBot allowed, GPTBot and Google-Extended blocked. A site with unique knowledge, for example paid analyses, may need stronger protection. A service business looking for clients? Usually it pays off more to be visible. But no illusions: letting a bot in gives you a chance to appear in answers, it doesn&#8217;t guarantee anything.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Not sure how your site is doing today? A <a href=\"https:\/\/www.web-systems.pl\/en\/seo-audits\/\">company website SEO audit<\/a> can help, and it also covers robots.txt and crawler access. In practice I recommend one thing: always check AI crawlers in robots.txt together with your server&#8217;s responses. Only then can you see what the bots actually get.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Frequently_asked_questions\"><\/span>Frequently asked questions<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n<div id=\"rank-math-faq\" class=\"rank-math-block\">\n<div class=\"rank-math-list \">\n<div id=\"faq-question-1792741741303-0\" class=\"rank-math-list-item\">\n<h3 class=\"rank-math-question \"><span class=\"ez-toc-section\" id=\"Will_blocking_GPTBot_remove_my_site_from_ChatGPT\"><\/span>Will blocking GPTBot remove my site from ChatGPT?<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<div class=\"rank-math-answer \">\n\n<p>No. OAI-SearchBot is responsible for showing sites in ChatGPT search. GPTBot only collects content for model training, so blocking it doesn&#8217;t affect search results.<\/p>\n\n<\/div>\n<\/div>\n<div id=\"faq-question-1792741741303-1\" class=\"rank-math-list-item\">\n<h3 class=\"rank-math-question \"><span class=\"ez-toc-section\" id=\"Will_blocking_Google-Extended_lower_my_rankings_in_Google\"><\/span>Will blocking Google-Extended lower my rankings in Google?<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<div class=\"rank-math-answer \">\n\n<p>No. According to Google&#8217;s documentation, this token doesn&#8217;t affect your presence in Search and isn&#8217;t a ranking signal. It only decides whether your content can be used to train Gemini models and for grounding.<\/p>\n\n<\/div>\n<\/div>\n<div id=\"faq-question-1792741741303-2\" class=\"rank-math-list-item\">\n<h3 class=\"rank-math-question \"><span class=\"ez-toc-section\" id=\"Does_ChatGPT-User_respect_robotstxt\"><\/span>Does ChatGPT-User respect robots.txt?<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<div class=\"rank-math-answer \">\n\n<p>OpenAI&#8217;s documentation notes that robots.txt rules may not apply to it, because it acts on a user&#8217;s request. And in general, robots.txt is not a security measure at all. Content that should be inaccessible is protected at the server level, for example with a login.<\/p>\n\n<\/div>\n<\/div>\n<\/div>\n<\/div>","protected":false},"excerpt":{"rendered":"<p>&#8220;AI crawlers&#8221; are really several different bots. The robots.txt file lets you decide separately whether your site&#8217;s content can be used to train models and whether your site should appear in AI search answers. Some people say &#8220;block AI&#8221;, others say &#8220;be visible in ChatGPT&#8221;, and in all that noise it&#8217;s easy to miss one [&hellip;]<\/p>\n","protected":false},"author":9,"featured_media":30855,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[822],"tags":[2534,2490,2530,1081,2533,1246],"class_list":["post-30863","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-seo-sem-en","tag-ai-crawlers","tag-chatgpt-x","tag-gptbot-x","tag-how-to","tag-robots-txt-x","tag-seo-en"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.web-systems.pl\/en\/wp-json\/wp\/v2\/posts\/30863","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.web-systems.pl\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.web-systems.pl\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.web-systems.pl\/en\/wp-json\/wp\/v2\/users\/9"}],"replies":[{"embeddable":true,"href":"https:\/\/www.web-systems.pl\/en\/wp-json\/wp\/v2\/comments?post=30863"}],"version-history":[{"count":2,"href":"https:\/\/www.web-systems.pl\/en\/wp-json\/wp\/v2\/posts\/30863\/revisions"}],"predecessor-version":[{"id":30912,"href":"https:\/\/www.web-systems.pl\/en\/wp-json\/wp\/v2\/posts\/30863\/revisions\/30912"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.web-systems.pl\/en\/wp-json\/wp\/v2\/media\/30855"}],"wp:attachment":[{"href":"https:\/\/www.web-systems.pl\/en\/wp-json\/wp\/v2\/media?parent=30863"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.web-systems.pl\/en\/wp-json\/wp\/v2\/categories?post=30863"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.web-systems.pl\/en\/wp-json\/wp\/v2\/tags?post=30863"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}