AI Chatbots: No Better for Medical Advice Than Internet Searches
People are increasingly turning to AI for health advice. However, a new study suggests this may not be helpful. Research published in Nature Medicine found AI chatbots did not help people make better medical decisions.
The study compared AI tools to traditional methods like internet searches. Participants using AI performed no better in choosing the correct course of action. This finding is significant as evidence on AI’s safety for medical guidance remains limited.
Researchers from the University of Oxford led the study. They designed ten medical scenarios with doctors. These ranged from mild illnesses to severe emergencies like brain hemorrhages.
A Gap Between Technical Ability and Real-World Use
First, the AI models were tested without human input. Three popular large language models correctly identified the medical condition 94.9% of the time. However, they selected the correct next step, like going to urgent care, only 56.3% of the time.
Then, 1,298 human participants were recruited. They assessed symptoms using AI, their own knowledge, standard searches, or the UK’s NHS website. The results were revealing.
Participants using any method identified the relevant condition in fewer than 34.5% of cases. They chose the correct action in less than 44.2% of cases. Critically, AI users showed no advantage over those using traditional information sources.
Why the Interaction Fails
The study highlighted a crucial disconnect. “The knowledge exists within AI systems,” said co-author Adam Mahdi. However, this information does not reliably translate into useful guidance during real interactions.
Researchers reviewed detailed conversations. They found users often gave incomplete or inaccurate symptom descriptions. Simultaneously, AI systems sometimes produce misleading or incorrect responses.
For example, one user describing a “worst headache ever” with a stiff neck was correctly told to go to the hospital. Another user with similar symptoms saying “terrible headache” was advised to lie down in a dark room.
The study calls for more research. The team plans to conduct similar studies in different countries and languages. They aim to see if AI performance changes over time or in different settings.
