Cloudflare now lets you block AI bots by type — here's the play
Cloudflare split AI crawlers into Search, Agent, and Training so you can allow some and block others. New defaults hit September 15. What store owners should do.
Cloudflare just gave every website owner — including anyone on the free plan — a switch to control AI bots by what they're actually doing. Not one blunt "block AI" toggle. Three categories: Search, Agent, and Training. If you run a store or a content site, this is the first time you can say yes to the traffic that sends you customers and no to the traffic that just eats your work. And on September 15, the defaults change whether you touch anything or not.
What actually happened
On July 1, Cloudflare rolled out new options to manage AI traffic by use case. Their three buckets:
- Search — a crawler that indexes your content "so it can answer questions about it later." This is what puts you in AI search results.
- Agent — "automated behavior that is acting, usually in real time, on a person's behalf" — a ChatGPT or Claude agent fetching your page to complete a task someone asked for right now.
- Training — "a crawler taking your content to train or fine-tune a model." Your data gets absorbed; you get nothing back.
The controls are live for all customers, free tier included. The bigger change is the default: starting September 15, 2026, new domains onboarding to Cloudflare will block Training and Agent bots by default on ad-displaying pages, while leaving Search allowed (per Help Net Security). Cloudflare's logic — on pages where human attention is the product, keep the bots that undercut it out.
Why it matters for your business
The instinct to "block all AI" is a mistake that costs you customers. Search bots are how your products show up when someone asks an assistant "where can I buy X in Phoenix." Block those and you delete yourself from the channel that's replacing Google for a growing slice of buyers. Agent traffic increasingly is a customer — a person's assistant checking your inventory or price on their behalf. You want that.
Training crawlers are the ones with a real argument for blocking: they take your catalog, your descriptions, your photos, and give nothing back. Now you can refuse them specifically without going dark to the rest.
The catch is the September 15 default. If you spin up a new domain and don't check the settings, agent traffic could get blocked on your ad pages — quietly cutting off assistant-driven shoppers. Defaults are decisions someone else made for you. Know what yours are.
Key takeaways
- Cloudflare now separates AI bots into Search, Agent, and Training — allow or block each independently, free plan included
- Search sends you customers; Agent traffic can *be* a customer; Training just absorbs your content — don't block them all the same
- From September 15, 2026, new domains block Training and Agent by default on ad pages — check your settings so you don't lose assistant-driven traffic
- "Block all AI" is the wrong default for anyone selling online
Not sure which bots are hitting your store — or what your current rules actually allow? We audit AI crawler access and set it up so search and agent traffic reach you while training crawlers don't. See how we work or send us your domain.
Sources: Cloudflare, Help Net Security.
- #cloudflare
- #ai-crawlers
- #seo
- #ecommerce
- #bot-management
Tommy Rush — Founder, Rush Commerce
Operator turned builder. 15+ years running operations — now shipping the systems businesses run on. More
Get The Rush Report weekly — one email, zero fluff.