Bravo Crawl Hunter – Block AI Crawlers, Spam Bots & Protect Content

Description

Bravo Crawl Hunter helps you block AI crawlers, AI scrapers, spam bots, unwanted bots, and other automated crawlers from your WordPress site.

Choose which known bots to allow or block, and decide whether they are asked politely through robots.txt or denied outright with an HTTP 403 response.

You can also monitor and review crawler activity from the Bravo Crawl Hunter >> Dashboard menu in your WordPress dashboard.

The plugin provides detailed crawler activity, including total requests, blocked requests, top crawlers, most targeted pages, and crawler categories.

Every crawler the plugin knows about comes from a curated registry organized by purpose, so you can understand what a bot is used for before deciding how to handle it.

🤖 Block AI crawlers and scrapers

Bravo Crawl Hunter includes a curated registry of AI crawlers, scrapers, AI assistants, and AI search bots from major AI companies and services.

AI Crawler bots are blocked by default, while AI assistants and AI search bots can be selected individually. You can block selected bots using robots.txt rules, or HTTP 403 hard blocking.

You can also add custom User-Agent strings when a bot or scraper is not included in the built-in registry.

🐺 A curated crawler registry

More than 75 known crawlers ship with the plugin, grouped into seven categories:

• AI Crawler – Crawlers that collect content for AI-related purposes, including GPTBot, CCBot, ClaudeBot, and Bytespider.
• AI Assistant – On-demand fetchers behind assistants and chat features, such as ChatGPT-User and Claude-User.
• AI Search – Crawlers used by AI-powered search and answer services.
• Search Engine Crawler – Traditional search engine crawlers such as Googlebot, Bingbot, and DuckDuckBot.
• SEO Tool – Crawlers used by third-party SEO analysis and ranking tools, including AhrefsBot and SemrushBot.
• Archiver – Crawlers used by web archiving services.
• Other Crawler – Other recognized automated traffic, such as monitoring services and link-preview bots.

Only the AI Crawler category is blocked out of the box. Search engines, SEO tools, archivers, and other crawlers stay untouched until you opt in, so search visibility never changes without your explicit choice.

🛡️ Two ways to control access

• robots.txt rules – Publishes Disallow rules for the crawlers you select. Well-behaved bots honor them, though not every bot does.
• Hard block (403) – Returns an HTTP 403 response when a selected crawler reaches WordPress, whether or not it follows robots.txt.

The two methods are independent. Use either one, or both together, from Settings → General Settings.

🔗 Covers WordPress content endpoints

Hard blocking applies to front-end requests as well as WordPress REST API and XML-RPC requests, helping prevent selected crawlers from accessing your content through alternate WordPress endpoints.

⚙️ Custom User-Agents

Bots and scrapers that are not in the registry can be blocked by adding their User-Agent string under Settings → General Settings.

Up to 50 entries are supported, each matched anywhere in the incoming User-Agent header and case-insensitively, and enforced with the same protection methods as the registry.

🚀 Works with caching and CDNs

Server-side detection runs when a request reaches WordPress. A page served entirely from a server cache, caching plugin, or CDN may never reach WordPress, so it cannot be classified that way.

For those requests, the optional JavaScript Detection feature records known crawlers that load a cached page containing the plugin’s lightweight detection script.

It is disabled by default and can be enabled from Settings → General Settings.

📊 Dashboard

The dashboard summarizes crawler activity over the date range you pick:

• Total, blocked, allowed, and unknown requests, plus the number of crawlers seen.
• Activity trends across the selected range.
• A breakdown by individual crawler and by crawler category.
• The pages receiving the most blocked crawler requests.
• Recent crawler activity.

📋 Crawler Logs

The Crawler Logs screen is a searchable, filterable, paginated history of classified requests. Each entry records the time, crawler, path, action, and user agent.

Logging is on by default and can be switched off under Settings → General Settings; dashboard totals and charts keep updating either way.

Log retention is capped by a maximum entry count under Settings → Advanced, and the oldest entries are removed first.

💾 Export, import, and data on uninstall

Settings can be exported to a JSON file and imported back, to keep a backup or move a configuration to another site, from Settings → Advanced.

The same screen lets you reset everything to the recommended defaults, and choose whether settings, logs, and stats are kept or removed if the plugin is deleted.