The harsh interface between patient care and automation led to a highly avoidable death

Summary

hub topic lead Richard Jones highlights an incident where the sepsis warning AI system failed to highlight a patient's deterioration and led to an avoidable death.

Content

I'll hide the location of this tragic story.

A busy nurse was doing her evening rounds. The ward was short on staff and so the nurse took some observations and put them on her uniform as a Post-It note. She'd enter the data later.

The patient had cancer and was heavily immunocompromised.

The nurse got back around to the patient and took further observations. She then went to enter them in the system.

The AI in the system had been trained to understand that two observations so close (in time) was an issue and so it ignored one. This meant it did not enter the details of the patient's vitals that showed the patient had an issue (sepsis). The patient was given an Amber alert status instead of a Red one. The next day the patient died.

The nurse was not at fault. You could argue the system was not at fault. However, it lacked 'real-world' experience of how nurses operate.

The learning point here? I'm not sure. Mindless reliance on systems to spot the things we miss is unhelpful but I have never regretted a conversation with a nurse regarding how they work and how they care.

About the Author

Richard Jones, Co-Founder of Mission10x for Healthcare. Founder/Re-founder of various AI and digital health companies.