ChatGPT Earns American Board Certification in Hand Surgery - 04/04/24

Doi : 10.1016/j.hansur.2024.101688

Diane Ghanem ^a,⁎ , Joseph E. Nassar ^b, Joseph El Bachour ^b, Tammam Hanna ^c
^a Department of Orthopaedic Surgery, The Johns Hopkins Hospital, Baltimore, MD, USA
^b Faculty of Medicine, American University of Beirut, Lebanon
^c Department of Orthopaedic Surgery and Rehabilitation, Texas Tech University Health Sciences Center, Lubbock, TX, USA

^⁎Corresponding author.

Sous presse. Épreuves corrigées par l'auteur. Disponible en ligne depuis le Thursday 04 April 2024

Abstract

Purpose

Artificial Intelligence (AI), and specifically ChatGPT, has shown potential in healthcare, yet its performance in specialized medical examinations such as the Orthopaedic Surgery In-Training Examination and European Board Hand Surgery diploma has been inconsistent. This study aims to evaluate the capability of ChatGPT-4 to pass the American Hand Surgery Certifying Examination.

Methods

ChatGPT-4 was tested on the 2019 American Society for Surgery of the Hand (ASSH) Self-Assessment Exam. All 200 questions available online (onlinecme.assh.org) were retrieved. All media-containing questions were flagged and carefully reviewed. Eight media-containing questions were excluded as they either relied purely on videos or could not be rationalized from the presented information. Descriptive statistics were used to summarize the performance (% correct) of ChatGPT-4. The ASSH report was used to compare ChatGPT-4’s performance to that of the 322 physicians who completed the 2019 ASSH self-assessment.

Results

ChatGPT-4 answered 192 questions with an overall score of 61.98%. Performance on media-containing questions was 55.56%, while on non-media questions it was 65.83%, with no statistical difference in performance based on media inclusion. Despite scoring below the average physician’s performance, ChatGPT-4 outperformed in the ‘vascular’ section with 81.82%. Its performance was lower in the ‘bone and joint’ (48.54%) and ‘neuromuscular’ (56.25%) sections.

Conclusions

ChatGPT-4 achieved a good overall score of 61.98%. This AI language model demonstrates significant capability in processing and answering specialized medical examination questions, albeit with room for improvement in areas requiring complex clinical judgment and nuanced interpretation. ChatGPT-4’s proficiency is influenced by the structure and language of the examination, with no replacement for the depth of trained medical specialists. This study underscores the supportive role of AI in medical education and clinical decision-making while highlighting the current limitations in nuanced fields such as hand surgery.

Le texte complet de cet article est disponible en PDF.

Keywords : Artificial intelligence, Language model, ChatGPT, Orthopaedic surgery, Hand surgery, Hand board examination