# Medical AI benefits vary based on user expertise MIT study: AI explanations can mislead non-experts - **Source:** [MIT News — AI](https://news.mit.edu/topic/artificial-intelligence2) - **Published:** 2026-08-04 - **Original article:** - **On Brifing:** https://brifing.si/s/e918b53e-5e34-44f5-8faf-7de0f6cbbf9f ## Groundbreaking research in AI-assisted medical diagnosis Researchers from MIT and other institutions revealed that user expertise level is critical in AI-assisted skin disease diagnosis. The study tested non-experts and primary care providers with different explainable AI systems. Results showed that while AI generally improved diagnostic accuracy, the impact of explanation methods varied by user knowledge level. Notably, large language model (LLM) explanations were found to carry risk of misleading non-experts, highlighting the need for tailored AI interfaces in healthcare settings. ## Non-experts show excessive trust in AI The study found that participants without medical expertise placed undue trust in AI systems. LLM-based explanations were accepted by non-experts regardless of their accuracy, with vague and generic explanations perceived as more convincing. This revealed that accuracy improvements among non-experts largely stemmed from uncritical deference to AI rather than genuine understanding. The findings underscore the risks of automation bias in medical AI applications for general users. ## Clinicians resist incorrect AI guidance One of the study's most striking findings was clinicians' resilience to incorrect AI assistance. Expert physicians performed best when receiving only the model's prediction without explanations. AI explanations provided minimal accuracy improvement for clinicians and sometimes decreased performance. Researchers attribute this to clinicians' ability to critically evaluate AI predictions against their professional training, demonstrating the value of medical expertise in AI-assisted diagnosis. ## Comparison of explainable AI methods The research tested four explainable AI approaches: prediction with confidence level only, prediction supported by similar images, heatmap highlighting important regions, and LLM-based plain language explanations. While all methods improved non-expert accuracy, LLM explanations showed the strongest effect but also the highest potential for misleading when incorrect. This tradeoff between explainability and potential for deception presents a key design challenge for medical AI systems targeting non-specialist users. ## Fairness-constrained model reduces skin tone bias The study employed an AI model specifically designed to combat bias against darker skin tones. This fairness-constrained model significantly improved diagnostic accuracy across all user groups and reduced skin tone-based diagnostic disparities. Particularly notable was its enhancement of non-experts' ability to correctly identify non-cancerous moles. These results demonstrate how addressing algorithmic bias can directly improve clinical outcomes in diverse patient populations. ## The danger of automation bias The study highlights automation bias as a critical risk in AI system design. As Marzyeh Ghassemi notes, both AI and explanation methods can trigger this bias in humans. The research particularly emphasizes that individuals with the least medical knowledge are most vulnerable to being misled by incorrect AI explanations. Researchers advocate for AI systems that actively encourage critical thinking rather than passive acceptance, especially in healthcare applications where errors can have serious consequences. ## AI must be personalized for user profiles Orson Xu from Columbia University points out that the same AI explanation could help an expert while misleading a novice. Xu argues that AI systems must consider user profiles during design, noting that those who might benefit most from AI assistance are often most susceptible to being misled by it. The research team emphasizes that how a recommendation is presented matters as much as its correctness, calling for more nuanced approaches to AI explanations in medical settings. ## Critical lessons for the future of medical AI Published in Nature Medicine, this study offers crucial insights for AI in healthcare. It demonstrates that one-size-fits-all AI approaches are inadequate, and systems must be tailored to user expertise levels. With many FDA-approved AI tools already in use for early diagnosis of skin conditions, these findings have immediate clinical relevance. The researchers make a compelling case that user-centered design is essential to realizing AI's full potential in medicine while mitigating risks of overreliance and automation bias. ## Sources covering this story - https://news.mit.edu/2026/medical-ai-assistance-benefits-vary-based-on-user-expertise-0804 — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._