AI versus MD: Evaluating the surgical decision-making accuracy of ChatGPT-4 (18 May 2024)

Summary

This study in Surgery aimed to investigate the accuracy of ChatGPT-4’s surgical decision-making compared with general surgery residents and attending surgeons. Five clinical scenarios were created from actual patient data based on common general surgery diagnoses. Scripts were developed to sequentially provide clinical information and ask decision-making questions. Responses to the prompts were scored based on a standardised rubric for a total of 50 points. Each clinical scenario was run through Chat GPT-4 and sent electronically to all general surgery residents and attendings at a single institution. Scores were compared using Wilcoxon rank sum tests.

The results showed that, when faced with surgical patient scenarios, ChatGPT-4 performed superior to junior residents and equivalent to senior residents and attendings. The authors argue that large language models, such as ChatGPT, may have the potential to be an educational resource for junior residents to develop surgical decision-making skills.