Prompt injection: #1 with experts, #12 in the incidents
A new study finds no detectable agreement between expert LLM risk rankings and the public incident record. Build your agent threat model from your own logs.
Two leaders of the OWASP Top 10 for LLM Applications project ran their own list against the public incident record and could not show the two agree. Prompt injection sits at #1 by expert judgment and #12 by incident count. If you have been using a top-10 list as your agent threat model, that is the finding to sit with — not because the list is wrong, but because a ranking and an incident count are measuring two different things.
What actually happened
Kyriakos "Rock" Lambros and Steve Wilson published an exploratory, non-peer-reviewed analysis to arXiv on August 18, covered by VentureBeat. They pulled 7,714 LLM security incidents from CVE, GitHub Security Advisories, OSV, and the AIAAIC AI-harm database, labeled 6,639 against a 20-entry taxonomy, and applied a Bayesian correction for classifier error. The OWASP 2026 ranking they compared against weighted roughly 29 expert respondents at 75% and incident data at 25%.
Agreement between the two came out at a Cohen's kappa of 0.20, with a 90% confidence interval from −0.16 to 0.57. The interval crosses zero, so chance agreement cannot be ruled out. The authors are explicit that this is not an official OWASP release and does not supersede the list.
Read the caveats before you read the headline. Classifier precision ranged from 93% down to 13% depending on category, and about 38% of the gold-set incidents were judged out of scope. The incident record is a noisy instrument, and prompt injection is exactly the attack it under-counts: it rarely produces a CVE, because nothing is technically broken.
Why AI agent risk ranking matters for your business
Neither signal is your threat model. Expert consensus tells you what sophisticated people fear. Incident counts tell you what got written down. Your actual exposure is which of your agents can spend money, send mail, or write to production.
Rank by blast radius, not by list position. Enumerate every agent, every tool it can call, every credential it holds. An agent with read-only catalog access and one with your payment API are not the same risk regardless of what ranks #1.
Instrument first. You cannot build an incident record you never logged. Log every tool call with inputs, outputs, and the identity it acted as. That log is the only ranking that reflects your business.
A low incident count can mean invisible, not rare. That is the honest reading of prompt injection at #12.
Key takeaways
- Exploratory arXiv analysis (Aug 18, 2026) by two OWASP LLM Top 10 leaders; not an official OWASP release
- 7,714 incidents pulled from CVE, GHSA, OSV, and AIAAIC; 6,639 labeled to a 20-entry taxonomy
- Cohen's kappa of 0.20, 90% CI −0.16 to 0.57 — agreement is not statistically distinguishable from chance
- Classifier precision ranged 93% to 13% by category; ~38% of the gold set was out of scope
- Rank your own agents by what they can touch, and log every tool call so you have a record to rank
Scoped permissions and an audit trail, from day one. We build agent systems where every tool call is logged, every credential is narrow, and the risky actions need a human. Talk to us about your agent stack.
Sources: VentureBeat.
- #ai-security
- #prompt-injection
- #owasp
- #ai-agents
- #risk
Tommy Rush — Founder, Rush Commerce
Operator turned builder. 15+ years running operations — now shipping the systems businesses run on. More
Get The Rush Report weekly — one email, zero fluff.
Keep reading
Ringg raises $10M: voice AI is really an outcome layer
Peak XV put $10M into Ringg, a voice AI startup running 20M call attempts a month. The lesson: buy the workflow engine, not the channel.
Read itOpenAI Jalapeño benchmarks: your token price has a new floor
OpenAI published the first Jalapeño inference benchmarks at Hot Chips 2026. Here is what a custom OpenAI inference chip does to what you pay per token.
Read it