Looking at the Safety of AI from a Systems Perspective: Two Healthcare Examples (8 August 2023)

Summary

While there is much potential and promise for the use of artificial intelligence in improving the safety and efficiency of health systems, this can at times be weakened by a narrow technology focus and by a lack of independent real-world evaluation. It should be expected that when AI is integrated into health systems, challenges to safety will emerge, some old, and some novel.

In this chapter of the book Safety in the Digital Age: Sociotechnical Perspectives on Algorithms and Machine Learning, Mark Sujan argues that to address these issues, a systems approach is needed for the design of AI from the outset. He draws on two examples to help illustrate these issues: Design of an autonomous infusion pump and Implementation of AI in an ambulance service call centre to detect out-of-hospital cardiac arrest.