AI explanations make non-experts trust wrong answers

Here is the finding that should change how you roll out an internal AI tool: AI explanations make non-experts trust wrong answers, while the same explanations make experts slightly worse. A study published in Nature Medicine on August 4 put both groups in front of the same diagnostic model, and the explanation layer — the thing every vendor sells as the trust feature — helped exactly the people who didn't need it.

What actually happened

The paper is "Divergent impacts of explainable AI for dermatological diagnosis on clinicians versus lay people", led by Xuhai "Orson" Xu at Columbia with Marzyeh Ghassemi at MIT and Roxana Daneshjou at Stanford. Per MIT News, the researchers ran two groups against four kinds of AI assistance: a prediction with a confidence score and no explanation, similar reference images, a heat map of the important regions, and a plain-language LLM explanation.

Lay participants judged whether skin moles were cancerous. Primary care clinicians did the harder task of producing a differential diagnosis.

The split:

• Non-experts got more accurate with every explanation format — but the gain came from deference. They followed the model whether it was right or wrong, and reported finding vague, generic explanations more convincing than specific ones.
• Clinicians did best with the bare prediction and no explanation at all. They weren't fooled by incorrect assistance. As Xu put it, when a clinician already holds a working diagnosis, "a bad explanation gets caught."

Ghassemi's framing is the one to keep: AI can raise performance, but it has to be weighed against deference that produces new errors.

Worth naming what we could not verify: the participant counts and per-condition accuracy figures aren't in the public coverage, and the full paper is paywalled. Treat the direction as solid and the magnitudes as unread.

Why automation bias matters for your business

You are not deploying AI to your best person. You are deploying it to the new hire, the weekend shift, the seasonal temp, and — if it's customer-facing — to someone with no training at all. That is the lay-person condition in this study, and the study says the explanation UI makes them defer harder.

Three design consequences we now apply by default:

Don't ship an explanation as a trust control. Prose that sounds reasonable is not evidence. If you need a control, ship a check against something verifiable — the actual inventory count, the actual invoice total, the actual policy text — and show the source record next to the answer.

Route by expertise, not by seniority of the request. Where the reviewer can't independently evaluate the output, the answer should be a suggestion that requires an explicit second step, not a filled-in field they can tab past. Where the reviewer is an expert, get out of the way and give them the raw prediction.

Measure override rate. If nobody on a workflow has disagreed with the AI in three weeks, you don't have a well-tuned model. You have a rubber stamp, and you won't find out which until a customer does.

Sources: Nature Medicine, MIT News.