Balancing promise and risk: AI hallucinations, confabulations and omissions in healthcare

Summary

Artificial intelligence (AI), particularly ambient voice technology (AVT), is already reshaping healthcare—enhancing patient care and reducing clinicians’ administrative burden. In the context of AVT, the technology offers meaningful benefits to both clinicians and service users. When considering digital clinical safety, the primary concern must always be the individual receiving care—the person in front of the clinician at their time of need.

With the promise of improved outcomes for service users comes a significant responsibility. While AI can enhance accuracy and efficiency, it is not infallible. In this blog, Ben Jeeves, a Chief Clinical Information Officer, discusses types of AI errors, in particular hallucinations, confabulations and omissions, and why they can pose real risks to patient safety if not mitigated effectively. 

Content

What is AVT, AI hallucinations, confabulations and omissions?

Ambient voice technology is a system that captures the spoken conversation between a healthcare professional and a service user. The captured audio is converted into a written transcript, which is then processed by AI to produce outputs that would otherwise need to be dictated or typed.

Hallucinations occur when AI generates information that is entirely made up, with no grounding in the source conversation or data. In a clinical context, this might look like an AI scribe recording a diagnostic test result that was never discussed or adding a medication that was not prescribed. The output has no basis in reality. An example recently reported was an AI tool that generated a set of false diagnoses for one patient that led to him being wrongly invited to a diabetes screening appointment.[1] 

Confabulations occur when AI starts with a real or partial detail and then overreaches, filling in gaps based on assumptions. For example, the system might infer that a patient smokes because they have a chronic cough and are in a certain age group, even though smoking status was never confirmed. The detail feels plausible but is speculative and unverified.

Omissions occur when the AI fails to capture important information. This could mean leaving out a mention of a severe allergy, missing safeguarding concerns raised during the conversation or failing to include a test result.

These are not hypothetical risks. AI outputs in healthcare have been shown to hallucinate medication dosages, omit key symptoms or insert invented follow-up instructions. When such errors make their way into official records, the consequences can be serious. This makes it vital to understand how AI can introduce subtle but significant errors.

Why hallucinations can create patient safety issues

While this may seem obvious, hallucinations and related errors threaten more than just the neatness of a clinical note, they can directly impact patient safety.

Errors also propagate easily. Once a false detail is entered into a medical record, it may be copied into referral letters, discharge summaries or future notes, amplifying its reach and impact.

Inaccurate data can lead to delayed or incorrect diagnoses, unnecessary appointments,[1] investigations or inappropriate treatments. In worst-case scenarios, this may result in harm that could have been avoided with accurate records.

Finally, omissions can also be equally dangerous. Failing to record an allergy, symptom or clinical red flag can expose patients to significant and immediate risk. Omissions are highlighted as a risk in recent NHS England guidance on the use of AI-enabled ambient scribing products in health and care settings.[2]

Clinician safeguards against hallucinations, confabulations and omissions

Although AI introduces new risks, applying appropriate and effective risk identification and mitigation processes, namely those in the Data Coordination Board and medical device regulation specifications, help to lower these risks to manageable levels.[3]

While many controls are available, one of the most critical mitigations against hallucinations, confabulations and omissions is clinician approval of the outputs generated by AVT systems. This human oversight is not only essential for ensuring clinical safety, but also plays a pivotal role in regulatory considerations. For now, the concept of a 'human in the loop' remains central to how such tools are assessed and classified under MDR regulations.

In the fullness of time we will see more 'technical' level mitigations, reducing clinician burden further. However, for the time being, training users to be aware that hallucinations, confabulations and omissions can occur and highlight where they are most likely to occur (although caution with this one as we don't want clinicians to only check higher risk areas…) remains essential.

The balance

Humans are not 100% accurate, 100% of the time. There are and have been omissions in clinical notes—sometimes whole clinical sessions haven't been recorded due to a plethora of reasons (the same reasons that should be in the hazard log for your AVT). So, with the potential for hallucinations, confabulations and omissions in AI-generated outputs, we must ask: Should we hold digital health technologies to a higher standard than human performance or is parity with clinicians a fair benchmark? And if so—how do we measure that benchmark objectively?

We are living through an exciting time in healthcare technology—one that brings unprecedented opportunities to learn and improve. But learning alone is not enough; it must be captured, shared and translated into safer practice. Above all, the true measure of success will be how effectively AI helps us improve patient safety while also improving health outcomes and enhancing the experience of care.

References

1. Nolan B. K health service AI tool generated a set of false diagnoses for one patient that led to him being wrongly invited to a diabetes screening appointment. Fortune, 20 July 2025.

2. NHS England. Guidance on the use of AI-enabled ambient scribing products in health and care settings. Last updated 29 April 2025.

3. NHS England. Clinical risk management standards.

Related reading on the hub

• Patient safety and the role of AI in a cautiously optimistic future: A blog by Ian Fearnley
• New AI system to identify patient safety issues announced: Patient Safety Learning’s initial reflections
• One size does not fit all. How AI and better data can help us embrace complexity in diagnosis and treatment
• From pain to progress: How NHS trusts are tackling the complaints crisis with AI

Further blogs from Ben

• Applying a robust approach to digital clinical safety in diagnosis

About the Author

Ben Jeeves is Chief Clinical Information Officer (CCIO) with Clinical Safety Officer (CSO) responsibilities at T-Pro, and remains clinically active as an Advanced Practice Physiotherapist in an integrated musculoskeletal (MSK) service at Midlands Partnership University NHS Foundation Trust. He is also the current Chair of the Digital Health Networks CSO Council.

Ben qualified as a physiotherapist in 2008 and went on to specialise in MSK, completing a Master’s in Sport & Exercise Medicine which led to an Advanced Practitioner role in an Emergency Department. He later moved into community MSK services, where he led the digital integration of MSK, podiatry, physiotherapy and community pain services. As part of this work, he undertook Clinical Safety Officer duties; experience that enabled him to secure his ACCIO post.

Passionate about the potential of digital to meet the future needs of healthcare, while recognising the daily challenges of digital transformation, Ben blends frontline clinical work with leadership in digital health, governance and safety. Outside of work, he particularly enjoys cycling, food, coffee, gardening, and time with his family.