Cloudflare-OpenAI pilot: ChatGPT search found pages crawlers missed

Cloudflare and OpenAI say their AI discovery pilot helped ChatGPT search find and index hundreds of millions of web pages that OpenAI's crawler had not found on its own. Cloudflare published the early results on October 15, 2026. The pilot started in July and covered only about 1% of Cloudflare's network. If a slice that small surfaced that many orphaned pages, a lot of the web is invisible to AI search. Some of it is probably your catalog.

What actually happened

Per Cloudflare's post:

• Discovery signals. Cloudflare's network sees which pages exist and get traffic. It passes signals to OpenAI's search crawler so it can find pages that link-following misses. OpenAI then evaluates and ranks those pages itself. Cloudflare does not decide what ranks.
• Waste filtering. The signals flag requests for files that do not exist, sites that redirect almost every URL to the homepage, and duplicate content, so the crawler can skip them.
• Change signals. The crawler gets told when a page changed and when it did not, so it stops downloading the same unchanged page.
• Harm flags. Individual bad pages, such as a fake login form, can be flagged without blocking a whole domain.
• Metadata, not content. Cloudflare says the research uses metadata about pages for search and discovery, not page content for model training. It also says a signal about a page does not grant permission to access it.

Cloudflare warns site owners may see more crawling in the short term while newly found pages get evaluated. TechTimes noted the companies have not yet reported whether any of this sends traffic back to the sites. That is the next step on their list.

Why AI discovery matters for your business

Classic crawlers find pages by following links. Stores are full of pages nobody links to: variant URLs, filtered collections, old SKUs still in stock, landing pages built for one ad campaign. Google has a sitemap for these. AI search has mostly had guesswork.

If you sit behind Cloudflare, that gap may close without you doing anything. That cuts both ways. Pages you forgot about can show up in ChatGPT answers, including the stale ones.

What we would do this week:

• Clean the orphans. Find live URLs with no internal links. Either link them properly or retire them with a real 404 or 301. Do not let a discontinued product page become an AI answer.
• Check your AI crawler settings. In Cloudflare, open AI Crawl Control and see which bots you allow. OpenAI documents its crawlers separately on its bots page: OAI-SearchBot for search, GPTBot for training. Allow search, decide on training.
• Watch the logs. Expect a crawl bump. Make sure your origin and rate limits can take it.
• Track referrals. Tag ChatGPT traffic in analytics now, so you have a baseline before discovery changes it.

Sources: Cloudflare blog, TechTimes, OpenAI crawler docs.