Direct answer
What percentage of B2B SaaS websites allow AI crawlers like GPTBot and PerplexityBot?
In Growthract's 2026 AI Search Readiness Benchmark, 99.0% of B2B SaaS companies with a determinable result allowed OAI-SearchBot, Claude-SearchBot and PerplexityBot in their public robots.txt — one company blocked all three (and every other major AI crawler), and a small number returned an undetermined result.
01
Blocking AI crawlers is rare in this sample — access is close to universal, not a common defensive posture.
02
The one company blocking all tested crawlers did so deliberately across the board, not selectively.
03
A small 'unknown' denominator exists for cases where robots.txt could not be conclusively fetched — those are excluded from the rate, not counted as blocked.
If you're worried your robots.txt is quietly blocking ChatGPT or Perplexity, the base rate says you're probably fine — but "probably" is worth confirming.
99.0%
allowed OAI-SearchBot, Claude-SearchBot and PerplexityBot alike
Across our 100-company sample, all three of these frequently-tracked assistant crawlers showed identical access rates: 97 allowed, 1 blocked, 2 undetermined — a 99.0% allowed rate of the known denominator for each.
The one exception
The single company blocking these three crawlers blocked every other major AI crawler we tested too — GPTBot, ChatGPT-User, Claude-User and Google-Extended included. It wasn't selective blocking of one system; it was a blanket policy. We cover that finding, and re-verify it live, in Only 1 of 100 B2B SaaS Companies Blocks Every Major AI Crawler.
What a 99% allow rate doesn't mean
Permission to crawl is not proof of crawling, and crawling is not proof of citation. This figure tells you almost no one is accidentally locking these systems out — it says nothing about whether they're actually visiting, or whether any of them choose to cite a given page.
Server-log verification of actual crawler activity is covered in Crawler Intelligence.
Where this fits in the full benchmark
The complete methodology, every schema and llms.txt finding, and the downloadable dataset are in 2026 AI Search Readiness Benchmark: 100 B2B SaaS Websites.
Confirm you're not the 1%
Verify your own crawler access, don't assume it.
Run Growthract's free diagnostic to confirm which AI crawlers can actually reach your site.
FAQs
Does a 99% allow rate mean AI crawler blocking isn't worth checking?
It's still worth a five-minute check. Base rates describe the sample, not your specific site — and the cost of checking is far lower than the cost of an undiscovered block.
Why were two results 'undetermined' instead of counted as blocked?
Robots.txt could not be conclusively fetched for those entries. Counting an inconclusive result as a block would overstate the finding, so it's excluded from the denominator instead.
Continue exploring
Related insights
Case study
An illustrative strategy showing how a Series B fintech could improve citation coverage from 10% to 34% across a controlled set of high-intent ChatGPT and Perplexity prompts.
See the case study →
Insight
In our 100-company AI Search Readiness Benchmark, exactly one B2B SaaS homepage disallows GPTBot, ClaudeBot, PerplexityBot and every other major AI crawler in robots.txt. Here's the real finding, re-verified live.
Read the insight →
Insight
Original research across 100 B2B SaaS websites on llms.txt adoption, AI crawler access, JSON-LD, schema markup, canonicals, and technical readiness.
Read the insight →