The Operating and Anesthetic Reference Assistant (OARA): A fine-tuned large language model for resident teaching - 02/07/24
Abstract |
Objective |
This study aimed to fine-tune a large language model (LLM) for domain-specific text generation in surgical and anesthesia residency education.
Summary background data |
With growing interest in artificial intelligence (AI) for medical training, the potential of LLMs to transform residency education is explored.
Methods |
The 7-billion parameter base model “Vicuna v1.5" was trained on 266,342 lines of text from 821 peer-reviewed documents. We evaluated the model with 150 surgical or anesthesia queries and assessed accuracy, token count, and inference speed across various reasoning tasks. Tests of significance were conducted using ANOVA and chi-square analysis.
Results |
Our model achieved 65.3% accuracy, excelling in surgical case-based tasks. We found no significant difference in accuracy between knowledge domains (P=0.081), though longer response generation demonstrated poorer accuracy, with significant accuracy variation based on output length (P = 0.002).
Conclusions |
LLMs show potential in enhancing residency education. Our model's efficiency and task-specific accuracy highlights such promise, though limits in parameter count diminishes accuracy of longer response generation. Our findings showcase how AI may be integrated effectively within future residency training.
Le texte complet de cet article est disponible en PDF.Highlights |
• | A large language model, OARA, was fine-tuned using surgery and anesthesia educational materials. |
• | Language models have the capacity to provide relevant information without access to external resources. |
• | OARA demonstrated overall accuracy in responses but varied based on the length of generated output. |
• | Potential for large language models as an adaptive resource in operating room and in resident didactics. |
• | Large language models are adept for various applications in adaptive and real-time surgical learning environments. |
Keywords : Large language model, Surgical education, Anesthesia education
Plan
Vol 234
P. 28-34 - août 2024 Retour au numéro
Article précédent
- Thank you artificial intelligence: Evidence-based just-in-time training via a large language model
- Tanna J. Boyer, Sally A. Mitchell
- Disparities in the treatment of primary hyperparathyroidism: A scoping review and conceptual model
- Alexander S. Chiu, Markayle Schears, Mary Hitchcock, Rebecca Sippel, Amy Kind
Bienvenue sur EM-consulte, la référence des professionnels de santé.
L’accès au texte intégral de cet article nécessite un abonnement.
Déjà abonné à cette revue ?