Cloudflare has rolled out new AI crawler management tools to all customers, including those on free plans. These controls let website owners categorise automated traffic by purpose: Search, Agent, and Training. It sounds like a sensible step forward in managing how AI systems interact with your site, and for most purposes, it is. However, from 15 September 2026, Cloudflare will automatically apply the strictest applicable rule to crawlers that serve more than one purpose, and this is where things get complicated for site owners.
Why This Matters
- Googlebot, Applebot, and Bingbot are at risk of being blocked. These crawlers are classified as multi-purpose, meaning they handle both search indexing and, in some cases, AI training. If your Cloudflare settings block Training crawlers, the new rule-conflict logic could apply that same block to the search crawlers you actually want on your site.
- The impact on search visibility could be significant. If Googlebot cannot crawl your site, your pages may drop out of Google’s index over time. For small and medium-sized businesses that rely on organic search traffic, this is not a minor inconvenience.
- Many site owners will not realise this has happened. Crawler blocks rarely throw up obvious errors. You might not notice a problem until you see a decline in rankings or traffic, by which point some damage may already be done.
What to Do Before 15 September
- Log into your Cloudflare dashboard and navigate to the AI crawler settings to review what rules are currently active.
- Check each category separately. Understand what you have set for Search, Agent, and Training crawlers, and make sure you are not inadvertently applying a blanket block.
- Do not block Training crawlers unless you are certain of the consequences. If you want to restrict AI training use of your content, explore more targeted options rather than a broad block that may catch legitimate search crawlers in the crossfire.
- Use Google Search Console to verify that Googlebot can still reach your pages. Check your crawl stats and coverage reports after any changes.
- Keep an eye on Cloudflare’s documentation, as the behaviour of these rules may be refined before or after the September deadline.
A Common Mistake to Avoid
The most common mistake is treating AI crawler controls as a simple on/off switch without considering which crawlers fall into each category. Blocking “AI Training” sounds straightforward, but because some major search engine bots are classified as multi-purpose, a Training block can have consequences that reach well beyond what you intended. Make sure you understand exactly which bots are affected before saving any changes to your settings.
How Kraken Dev Co Can Help
If you are unsure how your current Cloudflare configuration might be affecting your site’s crawlability or search performance, we are happy to take a look. We work with businesses to review their infrastructure and security settings so that technical decisions do not unintentionally undermine visibility or performance. Sometimes a second pair of eyes on your setup is all it takes to catch something before it becomes a problem.
