Designing AI with patient safety at its core: Reflections from the Patient Safety Forum 2026

Summary

Last month, Public Policy Projects hosted their annual Patient Safety Forum in partnership with Patient Safety Learning. Held at the Royal College of Surgeons of England in London, it was attended by senior healthcare leaders, patient safety experts, representatives from the HealthTech industry, frontline healthcare professionals and patients. 

In this article, Patient Safety Learning reflects on one of the panel discussions—AI for patient safety: Innovation, assurance and strengthening communication.

Content

From AI-enabled ambient scribing tools that reduce the burden of administration, to predictive systems capable of detecting early warning signs before harm occurs, AI has significant potential to improve patient care and outcomes. Yet, alongside these benefits come risks—algorithmic errors, data bias, and challenges in maintaining trust, governance and oversight. 

At the Patient Safety Forum 2026 an expert panel was convened to discuss this topic, with the following members:

• Clive Flashman, Chief Digital Officer, Patient Safety Learning
• Dr Alison Cave, Chief Safety Officer, Medicines and Healthcare products Regulatory Agency (MHRA)
• Anil Mistry, AI Safety Lead, Guy’s and St Thomas’ NHS Foundation Trust
• Dr Basil Bekdash, Clinical Safety Officer, Sheffield Children’s NHS Foundation Trust
• Aleksander Alski, Head of Region – USA, Canada and UK, Vasco Electronics 

Panellists had a lively discussion with each other and the audience about how to balance innovation with assurance, to ensure that the use of AI in healthcare enhances safety rather than undermines it. They spoke about how AI should be understood as a support tool for healthcare professionals—it provides information and removes burden but, ultimately, staff treat patients.

In this blog we highlight several key topics that emerged from this debate.

Importance of patient safety

A key theme running throughout the panel’s discussion was the importance of patient safety being built into AI development at the outset.

Clive Flashman from Patient Safety Learning reflected on this point, suggesting that too often this is seen as a compliance ‘tick box’ or treated as an afterthought. Speaking to digital innovators, his message was that “you need to think about this from the very start when you are conceptualising the product”.

Panellists also recognised that putting safety at the centre of discussions around AI and healthcare means involving all stakeholders, not just the healthcare professionals using these technologies but suppliers too. Alexander Alski from Vasco Electronics emphasised the importance of this being an area of shared responsibility between suppliers and healthcare providers.

Getting regulation right

Alison Cave from the MHRA spoke about the ongoing work of the National Commission into the Regulation of AI. This Commission was established by the MHRA to review current regulations and provide recommendations for a new regulatory framework for AI in healthcare. It held a public call for evidence which Patient Safety Learning responded to earlier this year.

Discussing how to approach future regulation, she highlighted the importance of ensuring that “the risk is associated with the decision, not the technology itself”. It was noted that in some cases there may be very complex pieces of software in use, but these may be making very low-risk decisions. Panellists underlined the importance of having a risk-proportionate regulatory framework to support safe innovation.

Predicting future harm

The potential to use AI to identify patient safety issues is understandably an area of significant interest. Last year the Department of Health and Social Care announced that it planned to develop a world-first artificial intelligence (AI) early warning system to automatically identify safety concerns across the NHS.

Panellists were asked to consider what examples they had seen of AI moving from reacting to incidents, to predicting and preventing future harm.

They spoke about the value of AI as a support tool for clinicians and more broadly how it might be used to identify emerging patient safety issues. Basil Bekdash from Sheffield Children’s NHS Foundation Trust spoke about work that had been trialled in this area, but noted that currently there have not been many examples where these have been proven on a significant scale, stating:

“None of them have really quite got to the point where they're proven in widespread deployment and so I'm not going to predict that's going to happen in the next five years.”

Tackling bias

While an AI tool may be safe when properly implemented and used by a well-trained healthcare professional, it could be potentially dangerous if such training and support is absent.

Panellists concurred that having appropriate training and tackling bias were issues of critical importance in ensuring the safety of AI in healthcare. In particularly they discussed risks presented by:

• Confirmation bias—healthcare professionals favouring AI outputs that align with their pre-existing view and overlooking signals that may challenge this.
• Automation bias—over-reliance on AI systems and accepting their recommendations without sufficient critical evaluation.

Alison Cave from the MHRA said that part of the training should be ensuring that healthcare professionals understand the devices they are using and where there are trade-offs between sensitivity and a specificity. Basil Bekdash from Sheffield Children’s NHS Foundation Trust noted the importance of having in mind the different levels of digital competence of staff, stating that when designing AI systems:

“It is best to test by using your least capable people who are the least digitally enabled and that's not a criticism that's just the reality of the normal spread of what people do, and their primary function is to look after patients.”

Transparency and patient communication

As use of AI grows in healthcare, it is vital that patients understand how this is being applied if they are to have confidence in its safety. Panellists discussed issues around how to inform patients when AI influences their care, particularly when it affects clinical judgments.

Anil Mistry from Guy’s and St Thomas’ NHS Foundation Trust suggested that:

“If the AI result is going to affect their patient’s care, and it's going to limit their access to finite resources like a waiting list or appointments or ICU beds, then absolutely have that sort of communication.”

However, he also spoke about some of the challenges this raises; for example, if a patient asked about whether AI has been used in their care. In practice this could cover a very broad range of areas, from the use of ambient scribes to take notes to tools that analyse images from scans. Panellists indicated that transparency needed to be balanced and proportionate to both the risk and impact on individual care.

Governance requirements

AI healthcare technologies have significant scope to evolve and change over time. When they iterate rapidly (with new versions being released at regular intervals) it can be difficult for existing governance frameworks, designed for other types of medical devices, to keep up.

Panellists discussed the importance of having flexibility to governance arrangements. There was the suggestion that lower risks tools (such as those in Class 1 for Medical Devices under the MHRA framework in the UK) should have greater flexibility, with higher levels of scrutiny reserved for decision-influencing tools.

It was also made clear that any new regulation will need to carefully consider the level of ongoing evaluation that will be required to account for these systems evolving and changing over time. This may be much longer than for other medical devices and change at significant pace. One audience member commented that with these tools becoming increasingly complex, in the future “realistically there is going to be a need for an AI tool that assesses AI tools”.

Panellists also considered how procurement processes could act as potential leverage mechanisms for AI technologies in healthcare. It was noted they offer the potential opportunity to embed the open standards we want to see being used by AI technologies in the earliest stages of their design, putting safety concerns at the centre of the product before it ever reaches patients.

Improving the quality of data

Data accuracy, completeness and representativeness is key to ensuring AI technologies work safely in health and care environments.

Panellists noted that poor foundational data standards undermine AI model training and lead to unreliable outputs. Their discussion reflected that a significant proportion of time is often spent on data cleaning before even applying AI. Improving this would have wider benefits for research, operational efficiency and public healthcare.

As we increase the use of AI health technologies, it is vital that we do not embed existing health inequalities. Following on from comments in an earlier session from Professor Bola Owolabi from the Care Quality Commission, Alison Cave from the MHRA noted a “perennial challenge in all of our areas is to ensure that the training data is representative”. Training data for AI systems must be representative of diverse populations and care settings.

Sharing insights from the frontline

If healthcare organisations, professionals and suppliers are to share responsibility for the safe implementation of AI technologies in healthcare, this must go hand in hand with shared learning. Panellists discussed the need for sustained and transparent feedback loops between suppliers, regulators and healthcare organisations. On this point an audience member asked:

“How do we ensure our learning keeps pace so that existing insight from frontline teams that really know the business can optimally inform the evolution of products, but without stifling the pace?”

Panellists highlighted the absence of standardised mechanisms for frontline staff to provide real-time, structured feedback to AI suppliers on safety issues. One proposed suggestion to this was the potential to mandate native feedback functionality within AI health technologies. This would mean that feedback mechanisms are built directly into the AI tool’s user interface and workflow, allowing those using them to provide input about the AI’s output without leaving the system. 

Find out more about the Patient Safety Forum 2026

You can read more about different discussions and panel sessions at this year’s event in the below:

• Patient voice, safety and the NHS 10 Year Plan: Reflections from the Patient Safety Forum 2026
• Safe systems, safe cultures: reflections from the Patient Safety Forum 2026