A 2015 CNN beat GPT-6 Astra and Claude at skin lesions

A ten-year-old image classifier just beat two frontier LLMs. Dermatologist Dr Magnus Lynch of Guy's Hospital ran a skin lesion classification test on 200 biopsy-proven lesions. A fine-tuned Inception v3, a CNN first published in 2015, picked the right diagnosis first 62% of the time. GPT-6 Astra and Claude Fable 5.1 each scored 42%. The lesson is not about medicine. It is about using a general chat model for a narrow classification job.

What actually happened

Lynch published the results on the Lynch Lab site on October 25. The setup:

• Data. 200 lesions drawn from MILK10k, a public ISIC dataset. All were biopsied and confirmed by histology: 100 that should not be discharged (melanoma, basal and squamous cell carcinoma, actinic keratosis) and 100 benign.
• LLMs. Both models got a close-up and a dermoscopic image, default settings, one run each. They chose one of six diagnoses or "unsure," plus a second choice.
• CNN. Inception v3 was fine-tuned on 68,011 ISIC Archive images on a single H100. Every image of the test lesions and the same patients was removed from training.

First-choice correct: 125/200 for the CNN, 84 for GPT-6 Astra, 85 for Claude Fable 5.1. In the top two choices: 173, 134 and 138.

The LLMs were safe in a useless way. They kept almost all dangerous lesions, but discharged only 32 and 22 of 100 benign ones. The CNN discharged 76. Asked as a patient "do I need to see a doctor?", both LLMs said yes almost every time, including for harmless lesions.

Lynch lists the limits himself: 200 lesions and 30 melanomas cannot rank close results; one run per LLM; hard cases only; the LLMs may have seen MILK10k in training; Inception v3 is a baseline, not state of the art. He also discloses that the analysis was drafted with Claude Code. Not peer reviewed.

Why it matters for your business: pick the narrow model

Small businesses do classification all day. Product photo to category. Return photo to "damaged or not." Invoice to GL code. Support email to queue. The fashionable move is to send all of it to the biggest LLM.

This test shows the failure mode. A general model hedges, and a hedge does not route anything. A small model trained on your own labeled examples is often more accurate, cheaper per call, and runs on one GPU or none.

Our rule: if the job has a fixed label set and you have a few thousand labeled examples, benchmark a fine-tuned small model against the LLM on your data before you commit. Hold out a test set the model never saw. Count the errors that cost money, not just accuracy.

Sources: Lynch Lab: LLMs vs a CNN on skin cancer.