AI Chatbots and Medical Advice: Why You Shouldn’t Trust Them Yet

The rise of artificial intelligence (AI) chatbots has sparked both excitement and concern, particularly when it comes to healthcare. While these tools offer potential benefits in accessibility and convenience, recent research reveals significant risks associated with relying on them for medical advice. Despite excelling on medical licensing exams, large language models (LLMs) powering these chatbots often provide inaccurate, inconsistent, and even dangerous information to users seeking help with their health concerns.

The Confidence Problem: When Wrong Answers Sound Right

A key issue with LLMs is their tendency to deliver incorrect information with the same level of confidence as accurate advice. Dr. Mahmud Omara, a research scientist at Mount Sinai Medical Center, explains, “A doctor who’s unsure will pause, hedge, order another test. An LLM delivers the wrong answer with the exact same confidence as the right one.” This can be particularly dangerous as it may lead individuals to forgo proper medical attention or follow harmful recommendations.

Falling for Misinformation: The Impact of Language

Studies have shown that LLMs are surprisingly susceptible to medical misinformation, especially when presented in formal, clinical language. Research published in The Lancet Digital Health found that chatbots were more likely to accept false claims when they were phrased like a doctor’s note compared to casual language. For example, a recommendation for “rectal garlic insertion for immune support” was accepted 46% of the time when presented in clinical terms, versus only 9% of the time when presented in a more informal style. This suggests that LLMs prioritize the sound of authority over the truth of a claim.

No Better Than a Basic Internet Search

A study published in Nature Medicine further demonstrated the limitations of LLMs in medical decision-making. Researchers found that using chatbots to identify health conditions and determine appropriate actions offered no greater insight than a traditional internet search. Participants using LLMs did not make better decisions than those relying on conventional methods. This is partly due to the fact that users may not know how to ask the right questions, and chatbot responses often contain a mix of helpful and harmful advice, making it demanding to discern the best course of action.

Potential for Harm and the Require for Caution

While AI chatbots can offer some potentially helpful recommendations, experts caution against relying on them for critical health decisions. Marvin Kopka, an AI researcher at the Technical University of Berlin, notes that individuals without medical expertise have “no way to judge whether the output they get is correct or not.” A chatbot might misdiagnose a severe headache as meningitis, or suggest a “wait-and-see” approach when immediate medical attention is required, potentially leading to dangerous consequences.

Future Applications: A Role Beyond Direct Patient Advice

Despite the current risks, researchers believe that AI chatbots still have a role to play in medicine. Whereas, their application may be more suited to tasks beyond providing direct advice to the public. As Dr. Omar suggests, chatbots may be valuable tools in other areas of healthcare, but “not in the way people are using them today.”

Key Takeaways

• AI chatbots are not yet reliable sources of medical advice.
• LLMs can confidently deliver inaccurate or dangerous information.
• The way information is presented (clinical vs. Casual language) significantly impacts an LLM’s response.
• Using chatbots for medical decisions is no better than a standard internet search.
• Individuals should exercise extreme caution and consult with qualified healthcare professionals for medical advice.