# The skin-cancer AI winner would have discharged 18 melanomas **Summary:** Lynch's Inception v3 beat GPT-6 Astra and Claude Fable 5.1 at diagnosis, but its preset threshold would have discharged 18 of 30 melanomas. I would evaluate safe discharge separately. - Canonical: https://markhuang.ai/news/skin-cancer-ai-winner-discharge-risk - Language: en - Author: [Mark Huang](https://markhuang.ai/about) - Published: 2026-10-25 - Section: News - Tags: Medical AI, Skin Cancer, Inception v3, Multimodal LLMs, AI Evaluation, Clinical Safety - Source: [Lynch Lab](https://lynchlab.net/llm-skin-cancer) - License: https://creativecommons.org/licenses/by-nc/4.0/ --- ![A dermatoscope examines an amber glass oval while another passes through an open translucent green gate](https://cdn.markhuang.ai/news/skin-cancer-ai-winner-discharge-risk/hero.webp) *Recognizing a pattern and deciding it needs no further review require separate evidence. Conceptual illustration.* In dermatologist Magnus Lynch's skin-cancer experiment, a fine-tuned Inception v3 named the correct diagnosis for 125 of 200 biopsy-confirmed lesions. GPT-6 Astra and Claude Fable 5.1 managed 84 and 85. His [October 25 report](https://lynchlab.net/llm-skin-cancer) gives me a good reason to consider a specialist vision model before reaching for a general-purpose LLM. I would hesitate before turning that win into a discharge feature. At its preset threshold, the convolutional neural network (CNN) would have discharged 18 of the 30 melanomas, compared with three and four for the LLMs. These numbers describe recommendations made in an image test. My read is that the diagnosis result earns the specialist further study. The discharge result needs its own evaluation. ## The comparison rewards preparation Lynch gave both LLMs a clinical close-up and a dermoscopic image, the magnified view used in skin examination. Each received the same prompt at default settings, choosing among six diagnoses or answering "unsure," then saying whether the lesion could be discharged without treatment, biopsy, or follow-up. The CNN had different preparation. Lynch reports fine-tuning it on 68,011 clinical and dermoscopic images, excluding the test lesions and all other photographs of the same lesions or patients from training and validation. His result compares a specialist trained for the job with general models prompted to do it. One objection in the [Hacker News discussion](https://news.ycombinator.com/item?id=50233779) is that a fine-tuned specialist beating general models on a tiny dataset should not be surprising. I agree with the limit behind that criticism. This experiment cannot establish which architecture is inherently better at medicine. It can challenge the assumption that a capable general model is the best starting point for this particular image task. For a team choosing what to investigate, that is useful. I'd want the specialist in the comparison. I'd also want to settle what the tool will do before declaring a winner. ## The errors carry different consequences A wrong diagnosis and an unsafe discharge are different failures. If a system mistakes one cancer for another but still recommends assessment, its label is wrong while its escalation decision may remain useful. A benign label can lead to a different decision. I would score the recommendation separately rather than infer it from diagnostic accuracy. Lynch's table makes that distinction concrete. At its preset threshold, the CNN retained 77 of the 100 lesions he classified as unsafe to discharge. The LLMs retained 97 and 96. The CNN also cleared 76 of the 100 safe lesions, compared with 32 and 22 for the LLMs. I can see the appeal of clearing more benign cases: a useful tool might reduce unnecessary follow-up. But I would count that benefit alongside the unsafe lesions it misses. The LLMs' more cautious results also leave substantial work for clinicians. The threshold deserves its own evidence. Lynch's table reports alternative CNN thresholds with only one or two melanomas discharged, but does not give the corresponding benign-discharge totals for those alternatives. I cannot judge the workload tradeoff from that table alone. I'd want both outcomes measured on new cases, using a decision rule fixed beforehand. ## The camera and the case mix matter The underlying [MILK10k dataset](https://doi.org/10.34970/648456) pairs clinical close-ups with dermoscopic images. Lynch's test uses a selected, biopsy-confirmed subset, split evenly between lesions he classifies as safe and unsafe to discharge. That gives the test reference diagnoses. Before carrying its results into a product, I would want evidence from the intended clinic population and its process for taking images. A [2020 BMJ systematic review](https://pmc.ncbi.nlm.nih.gov/articles/PMC7190019/) of nine studies of skin-cancer smartphone apps found problems including selective recruitment and images chosen and acquired by clinicians rather than app users. Those were older apps, not the systems Lynch tested. That history makes me ask who selects the lesion and takes the photograph in the proposed product, and whether the evaluation reflects those conditions. Lynch also acknowledges the small sample and that each LLM case ran only once. With 30 melanomas, I would be careful about close rankings. The [cases, prompts, replies, and scoring code](https://d2gaqzonlm544h.cloudfront.net/skin-ai/data/llm-skin-cancer-test.zip) are available to inspect. That makes the experiment useful to question and extend, while leaving clinical performance unresolved. ## The result I would ask for next Before a pilot, I'd specify whether the tool proposes a diagnosis for review or recommends that someone needs no further care. Then I would evaluate that role using the images it will actually receive. For discharge, I would ask for missed unsafe lesions and benign follow-up together, with uncertain cases counted explicitly. This is the distinction I keep returning to in my writing on [what AI benchmarks measure](https://markhuang.ai/blog/what-ai-benchmarks-actually-measure): the task being scored has to match the decision being delegated. Different model errors give me a reason to test a combined system, too, but this experiment did not evaluate one. I would take Lynch's result as a reason to test a specialist, including a modest one. I would withhold autonomous discharge on this evidence. The most useful next result would show how much work the system can safely remove from clinical review.