Fake AI crawlers hunt .env files. User-agents aren't auth.

Somebody figured out that "AI crawler" has become a magic word on the modern web. GreyNoise reported a cluster of scanners forging the user-agent strings of well-known AI crawlers and using that cover to request the exact files where a misconfigured server leaks its keys. The fake AI crawlers are not indexing anything. They are asking your web root for /.env.

What actually happened

GreyNoise traced the activity to 824 IP addresses spread across 795 separate /24 networks — deliberately dispersed, so no single netblock ban catches it. The forged names belonged to Anthropic (ClaudeBot), OpenAI (GPTBot), Google, Perplexity, and Amazon. GreyNoise checked all 824 addresses against the published IP ranges those companies document for their real crawlers. Not one matched (GreyNoise).

The request pattern is the giveaway. The traffic went after /.env and its variants (.env.local, .env.production, .env.bak), /.aws/credentials, /.git/config, private key files, and password stores. And the forged crawlers never requested /robots.txt — while Anthropic's real crawler requested it more often than any other path (Help Net Security).

That last detail is the whole tell. A crawler that skips robots.txt and goes straight for your credentials file is not a crawler.

Why fake AI crawlers matter for your business

Over the past two years a lot of teams bolted AI-crawler handling onto their edge — allow this bot, block that one, exempt these user-agents from rate limits so the "good" AI traffic gets through. Those rules are almost always written against the user-agent string, because that is the field that is trivially available in a WAF rule builder.

A user-agent is a string the client chooses. It has never been authentication, and now there is a live campaign built specifically on the assumption that you treated it as one. If your edge exempts ClaudeBot from rate limiting, you built a fast lane for whoever types those nine characters.

Three things, in order:

Alert on the paths, not the bot. No legitimate crawler has a reason to ask for /.env, /.aws/credentials, or /.git/config. Any request for those is either an attack or your own scanner, and your own scanner should already be on a list you maintain. This alert is cheap and it has almost no false positives.

Verify by IP, not by string. Anthropic, OpenAI, Google, Perplexity and Amazon all publish address ranges for their crawlers. If you have crawler-specific rules, resolve them against those ranges. If that is too much plumbing to maintain, delete the exemptions instead — a rule you cannot verify is worse than no rule.

Then stop serving the files at all. Nothing above matters if .env is not reachable from the document root. Deployments that copy the whole repo into the web directory are how a .git/config ends up publicly readable. That is a five-minute fix in your deploy script and it closes the class, not the instance.

Sources: GreyNoise, Help Net Security.