Is It Really OAI-SearchBot? How to Verify OpenAI Crawler Traffic

A request in your server log can call itself OAI-SearchBot, GPTBot or almost anything else. The user-agent field is supplied by the requester, so seeing a familiar crawler name is the start of verification—not proof that the request came from the company named in that string.

That distinction matters for small-business websites. A genuine search crawler may help eligible pages appear in AI-assisted search, while an impersonator may be scraping content, probing URLs or consuming server resources. The right response is not to allow or block every request with one broad rule. It is to identify the crawler’s purpose, verify its network origin, then apply a policy appropriate to that crawler.

The three questions to answer

When I review suspected AI crawler traffic, I separate the investigation into three questions: What identity does the request claim? Does the source address match the crawler operator’s current published ranges or a trusted verified-bot signal? What did the request actually do—paths fetched, status codes received, rate and timing?

A user-agent match without network verification is weak evidence. An IP-range match without looking at behaviour can also be misleading if a firewall rule is stale or a log records a proxy address instead of the original client. Good crawler detection uses both identity and request evidence.

Know which OpenAI crawler you are looking at

OpenAI’s current crawler documentation distinguishes OAI-SearchBot from GPTBot. The official crawler overview explains each purpose and publishes its associated IP-range file.

OAI-SearchBot

OAI-SearchBot is used for ChatGPT search features. OpenAI recommends allowing it in robots.txt and allowing its published searchbot IP ranges if you want the site to be eligible for inclusion. Eligibility is not a ranking promise: being crawlable does not guarantee that a page will be selected or cited.

GPTBot

GPTBot is associated with crawling content that may be used to improve and train generative AI foundation models. Its robots.txt control is independent from OAI-SearchBot. A business can allow search crawling while indicating that GPTBot should not crawl for training.

If your policy decision is the main question, start with Webcurry’s separate guide: Allow ChatGPT Search but Block AI Training? A Small-Business Guide.

Step 1: preserve the right log fields

Before changing a firewall rule, export a small, time-bounded sample. Keep the timestamp with timezone, source IP, HTTP method, hostname, request path, query string, response status, bytes sent, request duration, user agent, referer when present, and any CDN or WAF action. Remove cookies, authorization headers and personal query values before sharing the sample.

If the site sits behind Cloudflare, Fastly, Akamai, a hosting proxy or a load balancer, confirm which field contains the original client IP. The web server’s remote address may be the proxy itself. Trust forwarded-IP headers only when they were inserted by infrastructure you control; public clients can forge an unprotected header.

Keep the sample narrow. Ten to fifty representative requests around the incident is usually more useful than exporting a month of unrelated visitor data.

Step 2: match the claimed crawler

Filter the logs for the relevant token, such as OAI-SearchBot or GPTBot. Do not search only for the word “bot”; that produces too much noise and can miss full browser-style user-agent strings.

grep -F "OAI-SearchBot" access.log

Treat this as a log-filter example, not a verification command. It only shows requests that claim that identity.

For each group, note the distinct source IPs, requested paths, response codes and request frequency. A genuine crawler can request robots.txt, sitemaps, HTML pages and page resources. A suspicious pattern might enumerate private endpoints, hit nonexistent paths at high volume, ignore crawl rules, or rotate through many addresses that do not belong to the operator’s published network.

Step 3: check the current published IP ranges

OpenAI publishes machine-readable network ranges for its crawlers. Use the file linked from the current official documentation for the exact crawler you are checking. For search crawling, that is searchbot.json; GPTBot has its own gptbot.json file.

Do not copy one IP range from a blog post into a permanent firewall rule. Provider infrastructure can change. Fetch the current official file, validate that it came from the expected HTTPS host, parse the CIDR ranges, and test whether each logged source IP falls inside at least one range.

https://openai.com/searchbot.json

https://openai.com/gptbot.json

A CIDR range represents a network, not one address. String-prefix comparisons are unreliable: an address that begins with the same characters may still be outside the network. Use your firewall, CDN, programming language or a maintained IP-address library that performs real CIDR membership checks.

If the address does not match the current published file, do not allow it merely because the user agent says OAI-SearchBot. Apply your normal bot, rate-limit and abuse controls. OpenAI support has likewise advised verifying claims against published ranges or a CDN’s verified-bot signal.

Step 4: avoid the proxy-address trap

Suppose every crawler request appears to come from the same CDN address. Comparing that address with the crawler list will fail because you are measuring the intermediary. Check the CDN’s request logs or trusted client-IP field instead.

The reverse mistake is trusting any X-Forwarded-For value sent over the public internet. Configure the origin to accept forwarding headers only from known proxy networks, then use the last trusted hop to recover the client. If you cannot establish that trust chain, ask the host or CDN to perform the verification at the edge.

Step 5: test robots.txt and the actual response

A verified crawler can still be unable to use the site. Request the live robots.txt file and inspect the rules for the exact user-agent token. Then test a representative public page through the same CDN and security path.

Look for 200 responses on public canonical pages and essential resources. A 401 can indicate authentication. A 403 often points to a firewall, bot-protection rule or security plugin. A 429 indicates throttling. Redirect loops, JavaScript challenges, CAPTCHA pages and geo restrictions can also prevent automated access.

If verified crawler requests receive 403 responses, work through WordPress Shows a 403 Forbidden Error—How to Find the Rule Blocking Access.

For broader crawler failures reported as 4xx responses, see Search Console Says “Blocked Due to Other 4xx Issue”—How to Fix It.

Step 6: allow narrowly and rate-limit intelligently

If you decide to allow a verified crawler, scope the rule to the current official ranges and the relevant public paths. Do not bypass authentication for private dashboards, customer accounts, checkout sessions, preview URLs or internal APIs. A crawler allow rule should not become a general security exemption.

Keep sensible request-rate and concurrency protections. If a verified crawler produces load, return stable cacheable pages, improve server capacity or apply a measured rate limit rather than serving an endless challenge page. Review 429 and 5xx responses after the change.

Never disable the WAF for the entire site to fix one crawler. Test a narrow rule, record why it exists, assign an owner, and schedule review because published network ranges and product behaviour can evolve.

What to check on Wix, WordPress and Shopify

Wix

Confirm the public page is indexable, robots.txt contains the intended policy, and no connected security layer or external proxy blocks the published crawler ranges. Wix manages core hosting, so use available analytics and edge/security logs rather than looking for an origin-server access file that the platform does not expose.

WordPress

Check the host access log, CDN/WAF events, security-plugin logs and caching layer. Confirm that a plugin has not generated a conflicting robots rule. If the crawler receives different HTML from normal visitors, inspect maintenance mode, bot protection, geo restrictions and full-page cache variants.

Shopify

Review the generated robots.txt policy and any custom robots.txt.liquid changes. Then inspect CDN, app and analytics evidence available to the store. Avoid changes that expose customer, checkout or administrative routes.

Measure access without claiming attribution

Server logs can show that a verified crawler fetched a URL. They cannot prove that the URL caused a future AI answer, citation, sale or lead. Keep crawler activity separate from referral traffic and conversion reporting.

In analytics, watch for referrals and landing-page engagement, but remember that many automated fetches do not execute JavaScript analytics. Log-based crawl counts and user-session counts answer different questions.

For the new Search Console AI reporting workflow, use Google Search Console’s Generative AI Report: How to Measure AI Overview and AI Mode Visibility.

A practical verification checklist

1. Capture a time-bounded log sample with the original client IP and response data.

2. Group requests by claimed crawler token and source address.

3. Retrieve the current official IP-range file for that crawler.

4. Perform CIDR membership checks, not text comparisons.

5. Use a trusted CDN verified-bot signal as additional evidence when available.

6. Review paths, rate, status codes and WAF actions for behavioural context.

7. Apply a narrow allow, block or rate-limit policy and protect private routes.

8. Recheck logs after the change and document the rule for future review.

Frequently asked questions

Can a fake bot use the OAI-SearchBot user agent?

Yes. A requester can copy a user-agent string. Verify the source against OpenAI’s current published ranges or a trusted verified-bot service before granting special access.

Does allowing OAI-SearchBot guarantee ChatGPT citations?

No. It can make eligible public content crawlable for ChatGPT search, but placement and citation are not guaranteed.

Can I allow OAI-SearchBot but block GPTBot?

Yes. OpenAI documents the two controls independently: OAI-SearchBot is for search visibility, while GPTBot relates to potential model-training use.

Should I verify bots with reverse DNS?

Use the verification method the operator officially publishes. For these OpenAI crawlers, the official documentation provides IP-range files. Do not invent a reverse-DNS rule when it is not the stated verification method.

How often should I update the IP allowlist?

Treat it as dynamic configuration. Retrieve the official file on a controlled schedule or review it regularly, validate changes and keep a rollback path. Do not hard-code ranges indefinitely.

Why do logs show my CDN IP instead of the crawler?

The origin server may record the proxy hop. Use the CDN’s logs or a trusted original-client-IP field configured only for requests from known proxy networks.

Should verified crawlers bypass every security rule?

No. Allow only what public crawling requires. Private pages, login areas, checkout sessions and internal APIs should retain normal access controls.

Can analytics measure AI crawler visits accurately?

Browser analytics often miss automated requests because crawlers may not execute JavaScript. Use access logs for crawl activity and analytics for human sessions and conversions.

Verification is a security decision, not a naming decision

The safest crawler policy starts with evidence. Identify the claimed agent, recover the real client address, compare it with the current official network list, review the requested paths and responses, then apply a narrow rule. That preserves legitimate discovery without turning a copied user-agent string into a security pass.