Are answers obtained from artificial intelligence models for information purposes repeatable? - 04/10/25
Highlights |
• | The repeatability of orthodontic responses generated by large language models (LLMs) over time is of significant importance. |
• | While ChatGPT-3.5 demonstrated the highest level of consistency, the Gemini models exhibited moderate repeatability. |
• | The temporal variability in model performance underscores the need for caution when utilizing AI tools in patient communication. |
Summary |
Introduction |
The objective of this study was to assess the repeatability of orthodontic responses generated by multiple large language models across repeated time points.
Methods |
This experimental study assessed the answers provided by ChatGPT-3.5, ChatGPT-4.0, Gemini, and Gemini-Advanced to 40 frequently asked orthodontic questions. Each model was prompted with the same questions at three time points (T0: day 0, T1: day 7, and T2: day 14). Two blinded orthodontic experts independently evaluated responses using a 3-point accuracy scale. Cohen's Kappa and ICC were applied to assess inter-rater agreement and repeatability, respectively. In addition, Friedman test with Bonferroni post-hoc analysis and Spearman correlation were used for temporal comparisons.
Results |
Cohen's Kappa values between raters ranged from 0.624 to 0.749, indicating substantial inter-rater agreement. ICC values for repeatability ranged from 0.666 (Gemini) to 0.960 (ChatGPT-3.5). Friedman test results revealed significant differences in model accuracy at T0 and T2 ( P < 0.001). Post-hoc analysis showed ChatGPT-3.5 differed significantly from Gemini and Gemini Advanced. Spearman correlations between time points were positive but weak (ρ = 0.284 to 0.383, P < 0.001).
Conclusions |
The study revealed statistically significant differences in repeatability among AI models. Despite high accuracy, some models exhibited limited consistency over time. These findings underscore the importance of evaluating both accuracy and temporal stability when integrating AI systems into clinical orthodontic communication.
El texto completo de este artículo está disponible en PDF.Keywords : Large language models, Acquiring knowledge, Repeatable
Esquema
Vol 24 - N° 1
Artículo 101071- mars 2026 Regresar al número
Artículo precedente
- Predicting treatment pathways in Class II malocclusion patients using machine learning: A comparative study of four algorithms for classifying camouflage, growth modulation, and surgical decisions
- Mukesh Kumar, Sumit Kumar, Malvika Agarwal, Ekta Yadav, Sougandhika Gandi
- Comparison of differential and conventional rapid maxillary expansion on upper airway dimensions in children with bilateral complete cleft lip and palate: A CBCT-based secondary analysis of a clinical trial
- Denise Caffer, Daniela Garib, Carolina Faber, Alexandre Meireles Borba, Luiz Volpato, Rita de Cássia Moura Carvalho Lauris, Araci Malagodi de Almeida, Rafael Guerra Lund
Bienvenido a EM-consulte, la referencia de los profesionales de la salud.
El acceso al texto completo de este artículo requiere una suscripción.
¿Ya suscrito a @@106933@@ revista ?