Sakana AI review agent: 70% of planted errors, 16% of real ones

An AI review agent that finds most errors in a test and few in the wild is a useful warning for anyone buying "AI review." On Oct. 8, a Sakana AI team posted Beyond Imitation, a paper on its Multi-Layered Review (MLR) system for checking research papers. On planted contradictions in a paper's core claims, MLR detects more than 70%. On real documented errors in withdrawn papers, it gets 16% exactly right. Both numbers are honest. The gap between them is the lesson.

What actually happened

From the paper on arXiv:

• The architecture: MLR reads the full manuscript in layers before it writes a review. An Appendix Agent on Claude Haiku 3.5 summarizes the details outside the main text. A Literature Review Agent and a Review Agent run on Claude Sonnet 4.
• The benchmark: The team took 257 published conference papers and inserted synthetic logical contradictions, for 1,164 test cases. MLR detected more than 70% of contradictions placed directly in core claims.
• The real-world test: On a set of withdrawn arXiv papers with documented errors, MLR scored 16.11% exact match and 26.07% on a looser "similar" match. That beat the other systems tested, but it is a much lower number.
• Prompt injection: Hidden instructions inside manuscripts tried to steer the review. MLR detected them in 8 of 50 cases.
• Cost: About $0.50 per review, per the paper.

Why it matters for your business

Swap "research paper" for "contract," "invoice batch" or "pull request" and this is the pitch you are hearing from AI review vendors. Three points carry over.

Synthetic tests flatter. Planted errors are cleaner than real ones. A vendor demo on seeded mistakes tells you little about your mistakes. Before you buy, collect 20 to 50 real errors your team caught last year and run the tool on those originals.

Read the whole thing first. MLR's design choice, understand the full document before judging it, is the right pattern. A review agent that only sees a diff or one page will miss errors that live in the appendix or the attachment.

Documents can talk back. If 42 of 50 injected instructions got past a careful reviewer, assume a vendor's invoice or a candidate's resume can steer your AI reviewer too. Keep a human sign-off on anything that moves money or makes a commitment.

At $0.50 a pass, AI review is cheap enough to run on everything. Treat it as a second reader that flags work, not as the approver.

Sources: arXiv: Beyond Imitation (abstract), arXiv: full paper.