Direct answer
How can I tell if my website is invisible to AI crawlers?
Ten observable warning signs point to an AI-crawler visibility problem: blocked or absent bot access in robots.txt, zero AI-crawler hits in server logs despite allowing access, a large gap between raw HTML and rendered content, no structured data, inconsistent entity naming, thin or templated content, no mentions in prompt tests across several weeks, no AI-referral traffic in analytics, orphaned or unlinked key pages, and stale or contradictory information across the site.
01
Most of these signs are diagnosable in under an hour with free tools you already have access to.
02
Absence of a signal (no log hits, no referral traffic) is itself diagnostic — don't wait for a clear failure to investigate.
03
These are warning signs of an access or clarity problem, not proof that any specific AI system has 'blacklisted' a site.
"Invisible to AI crawlers" usually isn't one dramatic failure. It's a handful of small, checkable gaps that compound.
Each sign below has a specific, free way to check it — most take under ten minutes. None of them, alone, proves a system has permanently excluded you; together, several of them is a real pattern worth acting on.
Absence is a signal too
Several of these checks look for something missing — zero log hits, zero referral sessions. Don't wait for an obvious error message; a quiet absence over several weeks is itself the finding.
1–3. Access & rendering signs
Robots.txt blocks or omits major AI crawlers
Free check: Visit yoursite.com/robots.txt directly and check for disallow rules covering GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot or Google-Extended.
Zero AI-crawler requests in server logs despite allowing access
Free check: Search recent server or CDN logs for those same crawler user-agent strings. Permission without any actual requests is worth investigating on its own.
A large gap between raw HTML and what a browser renders
Free check: Fetch the page with curl (or 'view source') and compare it to the fully rendered page — if key facts only appear after JavaScript runs, some crawlers may never see them.
4–6. Structure & clarity signs
No structured data anywhere on the site
Free check: View page source on your homepage and a key product page and search for 'application/ld+json' — or run Google's Rich Results Test.
Inconsistent entity naming across your own properties
Free check: Compare your company name, tagline and category description across your homepage, LinkedIn page and any major directory listing side by side.
Thin or templated content with no direct answers
Free check: Read your three most important pages as a stranger would — does each one state its core answer in the first two paragraphs, or bury it under general framing?
7–8. Measured visibility signs
No brand mentions across repeated prompt tests
Free check: Run the same category and comparison prompts in ChatGPT and Perplexity weekly for a month — see our 12-prompt test for the exact list to use.
No AI-referral sessions showing up in GA4
Free check: Check GA4's traffic acquisition report for sessions from chatgpt.com, perplexity.ai or similar referrer domains over the last 90 days.
The prompt list for sign 7 is in 12 ChatGPT & Perplexity Prompts to Audit Your AI Search Visibility, and GA4 setup for sign 8 is in How to Track ChatGPT Referral Traffic in GA4.
9–10. Content hygiene signs
Important pages are orphaned or barely linked internally
Free check: Run a crawl of your own site (or check Search Console's Pages report) for key pages with few or zero internal links pointing to them.
Pricing or feature claims contradict each other across pages
Free check: Pull up your pricing page, your homepage and your most recent relevant blog post side by side and check whether the numbers and claims actually agree.
What a positive result here doesn't prove
Finding several of these signs means there's a real, fixable access or clarity problem worth prioritizing. It does not mean a specific AI system has permanently blacklisted your brand — public-web presence and AI-visibility behavior are related but distinct, and a single point-in-time absence isn't proof of a permanent one.
Found two or more of these signs?
Get a full diagnostic instead of checking each one by hand.
Growthract's free diagnostic checks crawler access, structured data, entity clarity and content structure in a single pass.
AI crawler visibility FAQs
How many of these signs mean I have a real problem?
There's no fixed threshold, but two or more — especially any combination of blocked crawler access plus zero log hits — is worth investigating immediately rather than waiting.
Can a site be crawled but still not cited?
Yes. Crawler access is necessary but not sufficient — relevance, content quality and competition for the same prompt all still factor into whether a system chooses to cite a page it can access.
Is there a single tool that checks all 10 signs at once?
Several of these checks require different tools (log files, GA4, manual review), which is exactly the gap a combined diagnostic like Growthract's is built to close.
Continue exploring
Related insights
Case study
An illustrative strategy showing how a Series B fintech could improve citation coverage from 10% to 34% across a controlled set of high-intent ChatGPT and Perplexity prompts.
See the case study →
Insight
In our 100-company AI Search Readiness Benchmark, exactly one B2B SaaS homepage disallows GPTBot, ClaudeBot, PerplexityBot and every other major AI crawler in robots.txt. Here's the real finding, re-verified live.
Read the insight →
Insight
Original benchmark data: 99% of sampled B2B SaaS companies allow major AI crawlers in robots.txt. Blocking is the rare exception, not the norm.
Read the insight →