UK health service AI tool generated a set of false diagnoses for one patient that led to him being wrongly invited to a diabetes screening appointment
AI use in healthcare has the potential to save time, money, and lives. But when technology that is known to occasionally lie is introduced into patient care, it also raises serious risks.
One London-based patient recently experienced just how serious those risks can be after receiving a letter inviting him to a diabetic eye screening—a standard annual check-up for people with diabetes in the UK. The problem: He had never been diagnosed with diabetes or shown any signs of the condition.
After opening the appointment letter late one evening, the patient, a healthy man in his mid-20’s, told Fortune he had briefly worried that he had been unknowingly diagnosed with the condition, before concluding the letter must just be an admin error. The next day, at a pre-scheduled routine blood test, a nurse questioned the diagnosis and, when the patient confirmed he wasn’t diabetic, the pair reviewed his medical history.
“He showed me the notes on the system, and they were AI-generated summaries. It was at that point I realized something weird was going on,” the patient, who asked for anonymity to discuss private health information, told Fortune.
After requesting and reviewing his medical records in full, the patient noticed the entry that had introduced the diabetes diagnosis was listed as a summary that had been “generated by Annie AI.” The record appeared around the same time he had attended the hospital for a severe case of tonsillitis. However, the record in question made no mention of tonsillitis. Instead, it said he had presented with chest pain and shortness of breath, attributed to a “likely angina due to coronary artery disease.” In reality, he had none of those symptoms.
A representative for the NHS, Dr. Matthew Noble, told Fortune the GP practice responsible for the oversight employs a “limited use of supervised AI” and the error was a “one-off case of human error.” He said that a medical summariser had initially spotted the mistake in the patient’s record but had been distracted and “inadvertently saved the original version rather than the updated version [they] had been working on.”
However, the fictitious AI-generated record appears to have had downstream consequences, with the patient’s invitation to attend a diabetic eye screening appointment presumedly based on the erroneous summary.
Source: Fortune, 20 July 2025