An evaluation framework for ambient digital scribing tools in clinical applications (13 June 2025)

Summary

Ambient digital scribing (ADS) tools alleviate clinician documentation burden, reducing burnout and enhancing efficiency. As AI-driven ADS tools integrate into clinical workflows, robust governance is essential for ethical and secure deployment.

Content

This study proposes a comprehensive ADS evaluation framework incorporating human evaluation, automated metrics, simulation testing, and large language models (LLMs) as evaluators. The framework assesses transcription, diarisation, and medical note generation across criteria such as fluency, completeness, and factuality. To demonstrate its effectiveness, the authors of the study developed an ADS tool and applied our framework to evaluate the tool’s performance on 40 real clinical visit recordings. The evaluation revealed strengths, such as fluency and clarity, but also highlighted weaknesses in factual accuracy and the ability to capture new medications. These findings underscore the value of structured ADS evaluation in improving healthcare delivery while emphasising the need for strong governance to ensure safe, ethical integration.

Related reading on the hub:

• Balancing promise and risk: AI hallucinations, confabulations and omissions in healthcare
• Patient safety and the role of AI in a cautiously optimistic future: A blog by Ian Fearnley
• New AI system to identify patient safety issues announced: Patient Safety Learning’s initial reflections
• One size does not fit all. How AI and better data can help us embrace complexity in diagnosis and treatment