Crawler Intelligence
Configuration says who can crawl. Logs show who did.
robots.txt tells you who's allowed in. Server logs tell you who actually showed up. Growthract keeps those two questions separate instead of blurring them together.
Configuration
robots.txt
Declared crawler access rules and directly observable configuration.
Observed activity
The short version
Allowed to crawl and did crawl are
two different facts.
See it live
Paste logs in. Get a real hit count out.
This is the actual log analyzer: paste a raw access-log window and it parses out recognized crawler hits and turns them into a chart.
Two evidence types
Permission and activity are not the same thing.
Crawler rules describe configuration. Server logs can describe observed requests. Growthract keeps those sources separate.
Crawler configuration
Inspect observable robots.txt rules and related technical controls.
Server logs
Analyze a user-supplied log window for recognized crawler requests.
Bot identification
Group recognizable AI crawler user agents such as GPTBot, ClaudeBot and PerplexityBot.
Observed activity
Describe what appears in the supplied logs rather than inferring activity outside that evidence window.
Browser-local analysis
Inspect logs without turning them into a new data pipeline.
Crawler log parsing is designed around user-supplied evidence. The analysis describes the supplied log window and does not require Growthract to claim continuous server monitoring.
Paste supported server logs, identify recognized crawler requests and inspect counts or activity from that evidence window. The result remains scoped to the logs you supplied.
Truth boundary
No crawler in one window does not mean no crawler ever.
Evidence boundary
Continue through the system
Technical Evidence
Website Audits
Inspect crawlability, indexability, structured data and page-level signals with deterministic checks wherever deterministic answers are possible.
Evidence Intelligence
Evidence
Keep observations, findings, recommendations and verification connected to the source evidence that supports them and the limitations that qualify them.
Search & AI Intelligence
Intelligence Overview
Bring technical, search, entity and AI evidence together into a persistent view of what deserves attention and what to do next.
See what Growthract already knows about your website.
No setup marathon. Point it at your site and see what is already there.
Analyze your website