AI HEALTH ADVICE: FAST BUT FATAL

James Klusener

@jamesklusener

 

Since the increase of artificial intelligence (AI) chatbots, such as ChatGPT and Gemini, along with AI overviews being baked right into Google search, students have turned to AI for health advice.  This idea may seem harmless on the surface. However, for several reasons, it’s best to avoid these platforms, as they can be misleading and also exploitative by harvesting data from you for their own benefit. 

 

While personal health data is protected under the Protection of Personal Information Act (POPIA) in South Africa, including data shared with AI chatbots, something known as ‘re-identification’, which includes using anonymous data to identify who it belongs to, is still worrying. 

 

Given how specific health data is, it can be easy in many cases to identify which sets of health data belong to which person, based on symptoms described, location, and previous histories that had been shared. This leads to concerns regarding privacy. These AI companies may also use all your data, everything you’ve input, chat history, and responses, to train their models further, which gives you very little control over where your data is going. 

 

Hallucinations are also causes for concern. When an AI chatbot hallucinates, it confidently gives you an objectively wrong answer. This is extremely dangerous, because as humans, we’ve learnt that confidence inspires accuracy, leading us to believe an AI chatbot when it lies. 

 

According to an article published by The Verge, using a chatbot for a medical diagnosis can lead to misinformation, causing users to either panic unnecessarily or, more dangerously, ignore symptoms that require professional medical attention. Furthermore, in an article from the BBC, Dr Rebecca Payne wrote, “Patients need to be aware that asking a large language model about their symptoms can be dangerous, giving wrong diagnoses and failing to recognise when urgent help is needed.” 

 

In addition, ChatGPT introduced its Health tab in January 2026, where you can upload fitness data, PDFs of medical records, etc. It’s clear that these companies have no problem with their services being used for health queries. 

 

Students in particular have found much use out of these chatbots. Many students simply do not have the funds to visit or consult a medical professional every time they may feel the need to. Getting a quick answer from a chatbot, such as ChatGPT or Gemini, was the obvious choice.

 

A student whose identity is known to Wapad made use of such chatbots when they underwent a minor surgery and made use of a chatbot to ask questions about their recovery and symptoms post-op. Other students have made use of AI for medical dosage advice for headaches or the flu. While some students have expressed concern, the overwhelming consensus is the ease of use, which outweighs the concerns for privacy. 

 

In a world where everything we do online is surveyed and harvested for the benefit of corporations, this disillusionment with privacy is easy to understand. 

A person holding a phone in front of their face (Source: HAI Stanford).

 

Edited by Isabel Burgers