Description
AI Crawlers Visibility, Analytics & Control is a WordPress plugin that helps site owners see how AI systems crawl their content, control access with robots.txt and PHP blocking, and keep a clear audit trail of bot traffic.
As AI search engines, chatbots, and training crawlers visit your site more often, you need visibility into who is accessing your pages and whether that access aligns with your content strategy. This plugin gives you a dedicated AI crawler analytics dashboard and practical bot control tools without replacing your existing SEO or security plugins.
Why monitor AI crawlers on WordPress?
AI companies use automated crawlers to index, train on, or retrieve website content. Common examples include GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), Bytespider (ByteDance), CCBot (Common Crawl), and many others.
With this plugin you can:
- See which AI bots visit your site and which pages they request.
- Compare traffic across date ranges and spot trends over time.
- Decide which crawlers to allow, discourage, or block.
- Keep logs for compliance, debugging, and content licensing decisions.
Key features
AI crawler analytics
- Live dashboard with total requests, unique crawlers, pages crawled, and blocked attempts.
- Charts for requests over time and traffic by crawler category.
- Top crawlers and most-crawled pages tables with period comparison.
- Recent activity log with filters for crawler, HTTP status, and content type.
- Blocked-attempt logging when PHP blocking is enabled.
68+ built-in AI crawlers
- Pre-configured detection for major AI, search, and scraper bots.
- Category labels (training, search, assistant, scraper, and more).
- Per-crawler info with user-agent pattern details.
- Enable or disable monitoring for individual crawlers.
Custom crawler detection
- Add your own user-agent patterns for bots not in the built-in list.
- Assign company name and category for consistent reporting.
robots.txt control for AI bots
- Add Disallow rules for selected AI crawlers.
- Rules are appended to your WordPress robots.txt output.
- Live preview before you publish changes.
PHP blocking (403)
- Hard-block selected AI crawlers before WordPress serves a response.
- Blocked requests are logged in the analytics dashboard.
- Works alongside robots.txt for stronger enforcement when needed.
Data management
- Configurable log retention (7 days to 1 year, or keep all data).
- Optional delete-all-data action from the admin.
- Delete plugin data on uninstall (optional setting).
Clean admin experience
- WordPress admin menu: AI Crawlers
- Analytics and AI Crawlers tabs with grouped side navigation.
- Built for site administrators — no code required.
Who is this plugin for?
- Bloggers and publishers who want to know when AI crawlers access their articles.
- Business and marketing sites tracking AI bot traffic to key landing pages.
- Developers and agencies managing crawler policy across client WordPress sites.
- Site owners exploring robots.txt and PHP options before broader AI visibility strategy.
Free vs premium
The free version includes crawler analytics, monitoring, robots.txt rules, PHP blocking, custom crawlers, and data retention controls. Premium features (AI Visibility Score, llms.txt, JSON-LD, email alerts, exports, and MCP integrations) are planned for a future release.
Installation
- Upload the
aI-visibility-suitefolder to/wp-content/plugins/, or install via the WordPress Plugins screen. - Activate AI Crawlers Visibility, Analytics & Control through the Plugins menu.
- Go to AI Crawlers AI Crawlers in the WordPress admin sidebar.
- Enable crawler tracking and choose your preferred data retention period.
- Open AI Crawlers Analytics to view traffic after AI bots visit your site.
FAQ
-
Which AI crawlers does this plugin detect?
-
The plugin ships with 68+ built-in crawlers including GPTBot, ClaudeBot, Google-Extended, Amazonbot, Bytespider, CCBot, PerplexityBot, Applebot-Extended, Meta-ExternalAgent, and more. You can also add custom user-agent patterns for bots not in the list.
-
Does this plugin block AI crawlers automatically?
-
No. Tracking is enabled when you turn it on, but blocking is opt-in. Use robots.txt rules to disallow crawlers in robots.txt, or PHP blocking to return a 403 response. Both can be configured per crawler from the AI Crawlers settings screen.
-
Will blocking AI crawlers hurt my SEO?
-
Blocking AI training or retrieval crawlers is separate from traditional search engine indexing. Disallowing GPTBot or similar bots in robots.txt does not block Googlebot or Bingbot by default. Review your robots.txt preview and allow crawlers you still want to reach your site.
-
Where do I see AI bot traffic?
-
Open AI Crawlers Analytics in your WordPress dashboard. You will see stats, charts, top crawlers, recent requests, and blocked attempts once AI crawlers visit your site and tracking is enabled.
-
Does the plugin modify my physical robots.txt file?
-
No. robots.txt rules are injected via the WordPress
robots_txtfilter, which is the standard approach on WordPress sites. Use the live preview in settings to see the effective output. -
Can I export crawler logs?
-
Export is planned for a future premium release. The current version stores logs in your WordPress database with configurable retention.
-
Is this plugin compatible with caching and security plugins?
-
Yes. Crawler detection runs on front-end requests. PHP blocking executes early in the request lifecycle. As with any blocking plugin, test on staging if you use aggressive caching or a reverse proxy.
Reviews
There are no reviews for this plugin.
Contributors & Developers
“AI Crawlers Visibility, Analytics & Control” is open source software. The following people have contributed to this plugin.
Contributors
Translate “AI Crawlers Visibility, Analytics & Control” into your language.
Interested in development?
Browse the code, check out the SVN repository, or subscribe to the development log by RSS.
Changelog
1.0.1
- Added crawler analytics dashboard with date ranges, filters, period comparison, and charts.
- Added AI crawler monitoring, custom crawlers, robots.txt rules, and PHP blocking (403).
- Added grouped side navigation for Analytics and AI Crawlers admin pages.
- Added miniOrange menu and navbar branding.
- Expanded built-in AI crawler list and crawler info tooltips.
1.0.0
- Initial release with MVC architecture and free/premium codebase structure.
- Added admin dashboard foundation.