Human versus AI scoring of surgical mental practice in the FingerFix Smart® procedure - 17/11/26

Abstract |
Introduction |
Assessment of surgical procedural knowledge remains largely dependent on human expertise, which may limit reproducibility and scalability. This study evaluated whether generative artificial intelligence could score a surgical mental practice task from audio transcripts after video-based learning of FingerFix Smart® dynamic external fixator placement.
Methods |
Thirty-six medical students who had completed a rotation in a hand surgery department watched a reference video performed by an expert surgeon. They then orally reproduced the procedure. The narrations were transcribed and assessed using a standardized 43-item checklist. Scores assigned by an expert human evaluator were compared with those generated by artificial intelligence using sequentially calibrated prompts.
Results |
The mean human score was 28.42 (SD 9.60) /43. The first prompt underestimated human scores, whereas the second overestimated them. After item-by-item discrepancy analysis, two additional prompts were generated. The fourth prompt showed the closest agreement with the human scoring, with a mean score of 29.33 (SD 8.20) out of 43, an absolute agreement intraclass correlation coefficient of 0.940 (95% CI, 0.884-0.969), and a mean bias of +0.92 points (95% CI, -0.09 to +1.93). However, the mean absolute error was 2.19 points, and the Bland–Altman 95% limits of agreement ranged from −4.94 to +6.78 points, indicating residual discrepancies at the individual level.
Conclusion |
Sequential prompt refinement reduced discrepancies between AI-generated and human scores within the development cohort, supporting the feasibility of iterative prompt calibration for the assessment of surgical procedural knowledge. However, because prompt development and assessment were performed using the same transcripts, these findings do not establish the reliability or generalizability of AI-based scoring. Independent validation is required.
Level of evidence |
III.
Le texte complet de cet article est disponible en PDF.Keywords : Artificial Intelligence, hand surgery, surgical education, mental practice, assessment, FingerFix Smart®
Bienvenue sur EM-consulte, la référence des professionnels de santé.
L’accès au texte intégral de cet article nécessite un abonnement.
Déjà abonné à cette revue ?
