PayPal AI assistant accuracy: 43% to 95% on the same model

PayPal pushed its internal AI assistant's accuracy from 43% to 95%, and the model never changed. It added the business knowledge the model was missing and kept that knowledge outside the model, VentureBeat reports. If your AI assistant gives confident wrong answers, a bigger model probably isn't the fix.

What actually happened

Ren Zhang, PayPal's VP and global head of AI product, described the work at VentureBeat events in October:

• The problem. The data science assistant already had clean, vetted data. It still got answers wrong. Ask about "new accounts" and it could pull the right records but use the wrong definition of "new."
• The fix. A knowledge layer with "golden queries" (the approved way to pull data for a given question) and analytical playbooks (how to investigate a question step by step).
• The result. 43% to 95% accuracy. "The LLM is still the same," Zhang said.
• Why it sits outside the model. So PayPal can plug in a better model later without rebuilding what it knows. Zhang also said frequent fine-tuning gets too costly as new models keep shipping.
• Usage. More than 1,000 questions and an estimated 1,600 hours saved over 10 weeks. Zhang cautioned that question volume alone doesn't prove productivity.

Deepika Khera, PayPal's senior director of AI, added two details. Every proposed agent decision is checked against a source of truth. And in some workflows the team forces the agent to consult memory at a set step instead of leaving it to the agent.

Why it matters for your business

The edge isn't the model. Everyone can rent the same one. The edge is the stuff only your team knows: what "active customer" means in your shop, which report the owner actually trusts, the order in which you check a refund.

Most small businesses keep that in people's heads. An AI assistant can't read heads. Write it down and you get a better assistant, plus a business that survives someone quitting.

It also keeps you portable. When a cheaper model ships next month, you swap one setting. Your definitions stay put.

Start here:

1. List your 20 most-asked internal questions and write the correct answer path for each.
2. Define your terms in one file. Customer, order, churn, margin.
3. Test on real questions from real users, not only the ones you expect. Zhang warned that test scores can hide what production users actually ask.

Sources: VentureBeat.