Can AI Crawlers Reach Your Site? A Verification Guide

Two requests hit the same URL within the same minute. The first carried an ordinary desktop browser user-agent: 200, a 57 KB body. The second carried GPTBot/1.2 — same address, same server: 403, 1.2 KB. What came back was not the page; it was the security layer's error screen.

This turned up on the site of an agency that sells GEO services. The irritating part is this: that site's robots.txt did not contain a single line blocking GPTBot. Quite the opposite — every AI crawler was listed one by one and every one of them had been given Allow: /. The file said "come in", the layer sitting in front of the server said "no", and that "no" showed up in no panel and in no report.

If you have been producing content, adding schema and fighting with page speed for months and you still never appear in answers, check this first. Everything else is built on top of it.

robots.txt does not grant access, it declares intent

robots.txt is not a contract, it is a declaration of intent. The Allow: / line you write into a text file does not mean the bot is able to connect to your server; it means that if the bot can connect, you consider it acceptable for it to crawl that path. Two separate layers are in play here:

• Declaration layer: robots.txt, llms.txt, noindex tags. The rules a bot is expected to obey. Compliance is voluntary.
• Enforcement layer: the server, the CDN, the firewall. The place that decides whether a connection is made at all. Compliance is mandatory, because the decision is already being taken there.

The most dangerous property of a "no" issued at the second layer is that it is silent. Put noindex on a page by mistake and Search Console tells you. If your server returns 403 to GPTBot, no warning appears anywhere — you simply never show up in the answers that bot feeds, and you put the reason down to content quality.

The block usually sits in one of these five places:

Hosting firewall. Shared hosting plans and many VPS panels ship with "bad bot" lists switched on by default. Those lists have been maintained for years and work on a logic that treats any user-agent they do not recognise as suspicious. New AI crawlers land on the "unknown crawler" side of most of them.

CDN bot management. Layers such as Cloudflare, Akamai and Fastly have bot fight / challenge modes. These modes automatically treat a client that cannot run JavaScript as suspicious. The majority of AI crawlers do not run JavaScript. Cloudflare additionally has a dedicated blocking switch for AI bots, and on some plans it can arrive switched on by default.

CMS security plugin. On the WordPress side, plugins such as Wordfence and iThemes carry their own bot rules. A plugin update can bring back a rule you never touched.

Rate limiting. Once a bot crosses a certain requests-per-minute threshold it gets a 429 or a 503. If it is crawling a large site, that threshold can be too low for AI crawlers.

Country or ASN block. A rule along the lines of "restrict traffic coming from outside Turkey" also cuts off all bot traffic arriving from cloud providers' data centres. AI crawlers come from data centres.

The five-minute test: ask directly with curl

Do not guess, measure. The test rests on a single principle: send a request to the same URL first with a normal browser user-agent, then with a bot user-agent, and compare the two answers. A bot response on its own tells you nothing; the comparison does.

Start with the reference (baseline) measurement. Behave like a normal browser:

curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
  -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0 Safari/537.36" \
  https://yoursite.com/important-page

The output comes back as something like 200 57342: the status code and the number of bytes downloaded. Those two numbers are your reference. Now try the same URL with six different bot identities:

for UA in \
  "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" \
  "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +claudebot@anthropic.com" \
  "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
  "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" \
  "Mozilla/5.0 (compatible; Google-Extended)" \
  "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
do
  printf "%-14s " "${UA:0:60}"
  curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -A "$UA" https://yoursite.com/important-page
done

Googlebot is the control group here. If Googlebot gets 200 while GPTBot gets 403, the block was not put in place "because it is a bot" but "according to which bot it is" — meaning somebody wrote a rule, and this is not a coincidence.

Run the test on an inner page you genuinely care about, not on the home page. Some rules apply only to particular directories; the home page can be open while everything under /services/ is closed.

Run the same loop against three more addresses: /robots.txt, /sitemap.xml and /llms.txt if you have one. These are the first files a bot visits. If your robots.txt returns 403 to the bot, every Allow line inside it is meaningless — the bot never managed to read the file at all. If the sitemap is closed, the bot cannot learn your page list either and moves only through the links it happens to find.

Reading the results

The sneakiest row is the third. You see 200 so you assume nothing is wrong; what actually reached the bot is a "we are verifying your browser" screen. This is exactly why you should measure not only the status code but the size as well. The %{size_download} inside -w exists for precisely this.

The test has two limits, and you should use it knowing both. First: an IP address that gets blocked once can stay on a blacklist for a while; if you fired off ten tests back to back, your later measurements may be measuring the block your own testing produced. Leave a few minutes between tests and, where possible, repeat from a different network. Second: you are not the real GPTBot. You are only sending a user-agent that carries that name. If the server performs reverse DNS verification, its behaviour towards the real bot may differ — which is why the curl test is a clue and the log record is proof.

Log verification: the proof is in the access record

Even if the curl test comes back clean, one question stays unanswered: did the bot actually turn up? Only the server access log can tell you that.

Three things are worth looking for in a log line: the client IP, the requested path, and the status code returned. Search for the bot user-agent and look at the codes on the lines that come back. If they are all 200, access is there. If there are 403s among them, look at which paths they fall on; a pattern usually emerges.

Where the log file lives: in cPanel, Metrics > Raw Access; in Plesk, Websites & Domains > Logs; in DirectAdmin, Site Summary / Logs. If you have direct access to Nginx or Apache, /var/log/nginx/access.log and /var/log/apache2/access.log are the standard paths. If you are on Cloudflare, the server log shows you Cloudflare's IP; to see the actual client you have to log the CF-Connecting-IP header. The AI Crawl Control screen in the Cloudflare panel separately shows how many requests each bot made and how many of them were blocked; we covered llms.txt and the bot management side in detail in our AI crawler management guide.

One warning: the user-agent field is free text, and anyone can write whatever they like into it. Not every "GPTBot" you see in the log is the real GPTBot. There are two ways to verify. Reverse DNS: resolve the IP with host <ip>; the domain that comes back should belong to the provider itself, then resolve that domain forward again and confirm it lands on the same IP. Official IP list: OpenAI, Anthropic and Google publish their crawler IP ranges as JSON on their own sites; check whether the IP in the log falls inside that range. If neither holds, that line is an impersonation, and probably traffic that should have been blocked anyway.

Full page in the browser, empty in the source

This is the second kind of silent loss. Status code 200, normal size, clean log — and yet the bot sees nothing on the page.

The reason is this: the content is produced client-side with JavaScript. You see a full page because your browser runs the JS; the bot gets an empty <div> because it does not run it. Googlebot renders up to a point, most AI crawlers do not.

Do the check with a single command:

curl -s -A "GPTBot/1.2" https://yoursite.com/important-page | grep -c "a-unique-sentence-from-the-page"

If it returns 0, the text is not in the HTML source. If it returns 1 or more, it is there. Repeat the same command for the page title, the main <h1> and, if you have one, a question from your FAQ block.

The fix is server-side rendering (SSR) or pre-rendering at build time (SSG): the HTML arrives from the server already full, and JS only adds interaction. This is a deliberate decision on our own site as well — real HTML is generated at build time for every route, because for a site that sells GEO, having its own pages readable by a bot is not up for negotiation. Getting the server and rendering decision right at the outset is the first item in a corporate website build. If your GEO investment has gone months without producing results, the first item in the diagnostic order in our three months and no results piece is exactly this.

When you remove the block: the cost is real

"Just open everything" is incomplete advice. The bill for bot traffic is real: bandwidth, server CPU, and on shared hosting the risk of hitting the resource limit. A bot that crawls aggressively can slow a small site down.

The balance is struck like this. Open the search and answer bots — OAI-SearchBot, PerplexityBot, Googlebot and the like arrive while producing an answer to a user's question and can bring you a citation. For training crawls the choice is yours — GPTBot and Google-Extended also use your content for model training; the price of shutting them out is that your brand appears nowhere in that model's general knowledge. Slow them down instead of blocking them: rather than closing the door completely, pull the rate limit down to a reasonable level, a few requests per second for example. The bot gets the content, the server stays up.

When you write the rule, target user-agents one at a time; broad rules such as "all unknown bots" automatically cut off the next new crawler as well, and you only notice months later.

What to tell your hosting provider

If you could not find a rule in the panel, open a support ticket. A vague request gets a vague answer. Give them these three components:

1. The measurement: "The same URL returns 200 and 57 KB with a normal browser user-agent; with the GPTBot/1.2 user-agent it returns 403 and 1.2 KB." Attach the command and the output as well.
2. The request: "Remove the block applied to these user-agents in the server firewall and in the mod_security rules: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended."
3. The evidence request: "Report which rule was triggered and at which layer (WAF, mod_security, bot list) it sits."

The third item matters: if you learn the name of the rule, you know what to look at when that same rule comes back with an update. To question whether the agency you work with performs this kind of technical verification, you can look at our agency evaluation questions.

Repeat this once a month

This is not a setup job, it is a maintenance job. Removing the block does not close the matter for good: a plugin update brings the old rule back, the CDN provider turns on a new default, the hosting company refreshes its bot list. None of them will tell you.

The monthly routine consists of three steps and genuinely takes five minutes: (1) run the curl loop above on three separate pages — the home page, a service page, a blog post; (2) save the output somewhere with a date attached, so that next month you can see the difference; (3) look at the number of 403s on the bot lines of the last 30 days in the access log, and it should be zero. If you want to read the results together with your answer traffic, the setup described in measuring AI traffic with GA4 completes this picture.

We published the result of the same test on our own site, with its date, in our GEO case study. If you run this test on your own site and get an output you cannot make sense of, send us the measurement — you can reach us from the contact page.

Frequently Asked Questions

If I have allowed AI bots in my robots.txt, is access guaranteed?

No. robots.txt is a declaration, not an enforcement. The moment a bot is able to connect to your server it reads that file and obeys the rules in it; but whether the connection gets made at all is decided by the server, the CDN and the firewall layer. A block placed at that layer turns the request away with a 403 without robots.txt ever coming into play, and that refusal produces no warning in any panel. This is why the permission declaration and the actual access have to be verified separately.

My request with a bot user-agent returned 200 but the size is very small, what does that mean?

Most likely the bot was served a verification or challenge screen rather than the real page. Because those screens are technically a successful HTTP response, the status code reads 200, but what sits inside is a JavaScript-based check instead of your page text, and a bot that does not run JavaScript sees no content at all. To tell them apart, compare against the size you got with a normal browser user-agent; if you are seeing a value between 1 and 15 KB instead of 57 KB, the content was not delivered.

How do I know the GPTBot record I see in the access log is genuine?

The user-agent field is free text and anyone can present themselves as GPTBot, so the name on its own is not proof. There are two verification methods: a reverse DNS query in which the IP address resolves to the provider's own domain and that domain resolves forward again to the same IP; or the IP address falling inside the official crawler IP ranges the provider publishes on its own site. If neither of them holds, that record is an impersonation and should not be counted as a real bot visit.

Will opening up all the AI bots harm my server?

The bandwidth and CPU cost of bot traffic is real, and on shared hosting in particular you can hit the resource limit. But the right answer is not to block them outright, it is to set a rate limit: a bot capped at a few requests per second still gets your content over time, and your server does not buckle under the load. Make the decision according to the type of bot; keeping open the search bots that arrive while producing an answer to a user's question brings you citations, whereas for crawling aimed at model training the choice is entirely yours.

How often do I need to repeat this test?

Once a month is enough, and in practice it takes five minutes. Repeating it is necessary because a block does not stay closed once it has been removed: a security plugin update can bring the old rule back, the CDN provider can switch on a new protection by default, the hosting company can refresh its bot list. None of these will send you a notification, so regular measurement is the only early warning mechanism you have.