AI is now the top risk to patient safety—what can organisations and patients do to protect themselves?

Summary

Clive Flashman is Chief Digital Officer at Patient Safety Learning. In this blog, he considers questions posed by participants at a recent panel discussion exploring the patient safety risks associated with the application of artificial intelligence (AI) technologies in healthcare. It covers:

• What boards need to understand before approving AI deployment.
• What safe AI looks like in practice.
• How governance and incident reporting need to evolve.
• What patients themselves can do to protect their own safety when AI is involved in their care.

Content

At the Digital Health Summer School 2026, I joined a panel discussion titled AI is now the top risk to patient safety—what can organisations and patients do to protect themselves?. In this discussion I sought to explain why unmanaged dependence on unmonitored AI tools is now one of the most pressing patient safety risks in the NHS. This blog reflects the questions I was asked (and many I wasn’t) and the answers I gave.

In one sentence, what is the greatest AI-related patient safety risk facing the NHS today?

It is unmanaged dependence on tools that no one is monitoring. We are deploying algorithms faster than we can record them. Accompanying this, we have no reliable way of managing model drift (the gradual decline in an AI model's performance over time because the data or environment it operates in has changed since it was developed and validated). Existing reporting systems were never designed to capture these kinds of issues.

Governance has not kept pace with adoption

AI adoption has clearly moved faster than organisational governance. Many trusts have no idea which staff are using which AI tools, and this is fundamentally a governance issue, not a technology issue.

What every board needs to understand before approving AI

Boards need training to fully understand the issues at stake, including:

• Approval has to keep pace with the evolution of the tools—it is not a ‘once and done’ decision. For example, an AI tool may, over time, change in terms of its capabilities, role and risk profile as it processes more data, significantly changing the way it operates. Boards may need to reassess AI tools when they undergo significant version upgrades or model changes.
• Accountability must be clear: who is the ultimate owner within the trust?
• Resilience planning: what happens if the tool fails, or the model drifts so far that it is no longer useful?

What does safe AI look like?

Safe AI should include the following characteristics:

• The AI models and versions are recorded in a digital asset register.
• Ownership is clear.
• The model has been validated on the relevant target populations, including those that are often underserved.
• The tool is monitored for drift and reported on when it goes wrong.
• It is removable if needed, without stopping the related clinical service(s).

AI as an IT issue, not a patient safety issue

Overall, NHS organisations are still treating AI as an IT issue rather than a patient safety issue. AI typically enters a trust via the digital and transformation teams. It is assessed for clinical safety via DCB0160, which is the clinical safety case that the NHS organisation has to write to assure the users of the digital technology (and its leadership team) that the digital solution is safe to use (from a clinical perspective). Subsequently, it might be reviewed by information governance and cyber security staff—but very rarely by patient safety staff. Every clinical AI tool or, better yet, every model, should have an assigned clinical owner.

Are we investing enough in clinical safety capability?

No. Most trusts are lucky to have one full-time equivalent clinical safety officer, and they are being asked to assess systems that are very different to what DCB0160 was originally written for. Static hazard logs simply do not work for adaptive systems. If we are spending billions on AI deployment, a proportionate slice of that investment should go towards expanding this valuable resource.

How should AI be assessed before deployment?

• On the relevant target patients, with their knowledge.
• Within the relevant workflows and pathways.
• With the existing quality of data the trust actually holds.
• Using more real-world evidence and fewer simulations.
• With test scripts that monitor whether AI outputs meet expectations.

Governance once AI is live

We should apply the same structure we would build for any high-risk clinical intervention:

• A named owner.
• Defined performance thresholds with automatic escalation when breached.
• Version logging—recording which model made which decision.
• Continuous or continued monitoring.
•  A rehearsed business continuity process.

Pharmacovigilance for algorithms

Pharmacovigilance is the science and practice of detecting, assessing, understanding and preventing adverse effects or any other problems related to medicines. Organisations should treat every model, and potentially each algorithm, in the same way we treat a medicine after licensing—in effect, pharmacovigilance for AI. This means:

• Continuous monitoring against agreed thresholds.
• Drift detection.
• Proper reporting of issues to the clinical owner and upwards as necessary.
• Post-implementation incident reporting.

How does incident reporting need to evolve?

Two main changes are needed:

1. Local risk management systems (LRMS) should be reconfigured to record which tool, version and model was involved in an incident, ideally pulled automatically from a digital asset register.
2. Nationally, the Learn from Patient Safety Events (LFPSE) service needs to evolve to reflect these types of incidents. Alongside this there should be a well-promoted, yellow card-type reporting route for patients, families and carers.

What should patients be told?

Patients should be given the answers to three questions, ideally before they even need to ask them:

• Is AI involved in my care?
• If so, what is it doing?
• Who do I speak to if I think it is getting things wrong?

Should patients be able to opt out of AI-driven care?

If the AI is a single tool making clinical decisions about a patient, then in principle, yes—an alternative should be offered where possible.

However this may not always be a straightforward case. For example, if AI is used via a triage algorithm (structured set of rules or decision logic used to assess a person's condition and determine the appropriate level of care or priority for treatment) embedded within a wider system. In such cases, trust staff may not know which parts are AI-driven, making it impossible to offer a clean opt-out. Martha’s Rule is a useful model here: perhaps there should be no automatic right to refuse the main system, but patients should always have the right to challenge it and ask for a relevant human expert to be involved.

Transparency: the coroner’s inquest test

Organisations should be transparent enough to survive detailed questioning at a coroner’s inquest. If a coroner asks about model versions, training data or performance monitoring, a trust should be able to answer this relatively quickly—and much of this information should be held in a digital asset register. For greater transparency, the case could be made that this information should also sit in the public domain.

Preventing AI from widening health inequalities

• At procurement, set out the relevant patient cohorts and ask to see the training data and performance metrics for each.
• Monitor stratified performance post-implementation.
• Recognise the risk of feedback failure—harm often concentrates in groups least likely to complain, so silence should never be mistaken for success. Actively seek feedback from those groups.

Balancing innovation with public confidence

Unsafe deployment damages public confidence, and lost confidence damages future innovation. What is needed is visible safety infrastructure—monitoring, reporting and accountability—combined with honesty when things go wrong. Build a safe governance system and confidence will follow, allowing innovation to thrive.

Practical steps for patients right now

If I were a patient undergoing complex treatment tomorrow, I would take five practical steps to reduce my AI-related risk:

1.  Ask the three questions: is AI involved in my care, what is it doing and who do I speak to if I think it's getting it wrong?
2.  Bring a second pair of eyes and ears—a family member, carer or friend.
3. ·Keep my own record of what I was told, when, by whom and what changed.
4. ·If I’ve used an AI chatbot before my appointment, record the exact query and answer given and share this with my clinician.
5. ·Know my escalation rights, such as Martha’s Rule.

About the author

Clive Flashman is Chief Digital Officer at Patient Safety Learning.