AI health warning as researchers say algorithms could discriminate against patients

Artificial intelligence in healthcare has left experts urging caution that a focus on predictive accuracy over treatment efficacy could lead to patient harm.

Researchers in the Netherlands warn that while AI-driven outcome prediction models (OPMs) are promising, they risk creating “self-fulfilling prophecies” due to biases in historical data.

OPMs utilise patient-specific information, including health history and lifestyle factors, to assist doctors in evaluating treatment options. AI’s ability to process this data in real time offers significant advantages for clinical decision making.

However, the researchers’ mathematical models demonstrate a potential downside, namely, if trained on data reflecting historical disparities in treatment or demographics, AI could perpetuate these inequalities, leading to suboptimal patient outcomes.

The study highlights the crucial role of human oversight in AI-driven healthcare. Researchers emphasise the “inherent importance” of applying “human reasoning” to AI’s decisions, ensuring that algorithmic predictions are critically evaluated and do not inadvertently reinforce existing biases.

The team then created mathematical scenarios to test how AI may harm patient health and suggest that these models “can lead to harm”.

“Many expect that by predicting patient-specific outcomes, these models have the potential to inform treatment decisions and they are frequently lauded as instruments for personalised, data-driven healthcare,” researchers said.

“We show, however, that using prediction models for decision making can lead to harm, even when the predictions exhibit good discrimination after deployment.

“These models are harmful self-fulfilling prophecies: their deployment harms a group of patients, but the worse outcome of these patients does not diminish the discrimination of the model.”

Read full story

Source: The Independent, 12 April 2025