Crawler
Which recognized AI crawler or user agent appears in the supplied logs.
Growthract Audit · Crawler Logs
See which AI crawlers actually visited your website.
Paste raw server logs and identify observed requests from recognized AI crawlers such as GPTBot, ClaudeBot and PerplexityBot — without uploading the raw logs to Growthract.
Open Crawler Logs
The analyzer is the Crawler evidence panel on the app’s Visibility page. Run a free audit on your site first to unlock it. No sign-in required.
Browser-local parsing Real server logs Observed requests
Real Crawler Logs interface
Permission vs observation
robots.txt and server logs answer different questions.
robots.txt
What crawlers are permitted to request.
robots.txt contains crawler-facing directives. An allowed crawler is not proof that the crawler has ever requested a page from your server.
server logs
What actually requested the server.
Access logs record requests represented by the server-log data you supply. They provide observable evidence of activity rather than permission alone.
Five-step workflow
From raw logs to crawler evidence.
No API connection is required. Supply the access-log data you want to inspect and let the browser-local parser identify recognized crawler activity.
Export your server logs
Get the access-log data from your hosting provider, reverse proxy, CDN or web server.
Paste the raw log data
Crawler Logs accepts Common Log Format, Combined Log Format and JSON Lines where supported by the parser.
Parse locally in your browser
The supplied raw logs are processed in the browser. Growthract does not upload your raw log file to its server for analysis.
Identify recognized AI crawlers
The tool looks for recognized crawler requests such as GPTBot, ClaudeBot, PerplexityBot and other supported AI crawler user agents.
Interpret the evidence
Use the observed requests to separate crawler permissions from crawler activity without assuming what happened after the request.
Supported inputs
Common Log FormatCombined Log FormatJSON Lines
Real product output
See recognized crawler activity in the supplied logs.
Crawler Logs surfaces the crawler evidence available in the data you pasted instead of converting it into a speculative visibility score.
- Recognized AI crawler names
- Observed request counts
- Activity represented by the supplied log data
- Evidence you can inspect directly
Crawler Logs result
What you can inspect
Keep the output tied to observable evidence.
Requests
How many matching requests appear in the portion of server-log data you supplied.
Requested URLs
Which website routes or resources were represented in those logged requests where available.
Observed activity
Evidence that a request reached the server represented by the supplied access logs.
Recognized crawler examples
A request is evidence of a request — nothing more.
OpenAI
GPTBot
A matching request in your supplied logs is evidence that the GPTBot user agent requested a resource represented in that log data.
Anthropic
ClaudeBot
A matching request shows observed ClaudeBot activity in the supplied server logs.
Perplexity
PerplexityBot
A matching request shows observed PerplexityBot activity in the supplied server logs.
Browser-local analysis
Raw logs stay in your browser.
Crawler Logs parses the raw log data locally in the browser. The supplied raw server logs are not uploaded to Growthract's server for processing.
As with any log-processing workflow, review your own internal data-handling requirements before copying sensitive operational data into any tool.
Evidence boundary
What a crawler log hit does — and does not prove.
Server-log evidence can support:
- That a request represented in the supplied log data occurred
- Which recognized crawler user agent made the request
- Which URLs were represented where the log includes them
- How many matching requests appear in the supplied data
- The distinction between allowed access and observed activity
A request does not prove:
- That the content was used for training
- That the page entered an AI index
- That retrieved information was used in an answer
- That the website will receive a citation
- That the brand will receive a recommendation
- That the website will improve its ranking
Related Growthract Audit capabilities
Connect crawler activity to the wider diagnostic.
AI Readability
Check the technical conditions available before crawler activity.
ExploreAI Search Visibility
Measure observable AI-search presence separately from crawler activity.
ExploreTechnical Methodology
See how Growthract separates deterministic evidence from inference.
ExploreAI Crawler Engineering
Explore the technical relationship between crawlers, entities and site architecture.
ExploreFrequently asked questions
Crawler Logs FAQ
What is an AI crawler log analyzer?
An AI crawler log analyzer reviews server access-log data for recognized AI crawler user agents. Growthract's Crawler Logs tool can identify requests from supported crawlers such as GPTBot, ClaudeBot and PerplexityBot in the logs you supply.
What log formats does Crawler Logs support?
Crawler Logs supports raw server access logs in Common Log Format, Combined Log Format and JSON Lines where supported by the parser.
Does Growthract upload my server logs?
No. Crawler Logs parses the supplied raw log data locally in your browser. The raw log file is not uploaded to Growthract's server for processing.
Is robots.txt proof that an AI crawler visited my site?
No. robots.txt expresses crawler permissions or restrictions. Server logs provide stronger evidence of what actually requested the server represented by those logs.
Does an AI crawler request mean my site was indexed?
No. A logged request proves only that the request represented in the supplied server logs occurred. It does not prove indexing, training, retrieval use, citation, ranking or recommendation.
Can crawler logs prove that ChatGPT or another AI system used my content?
No. Server logs can show recognized crawler requests, but they do not reveal how the retrieved information was later processed, stored or used by an independent AI provider.
Explore related resources