Objective: AI-derived language models are booming, and their place in medicine is undefined. The aim of our study is to compare responses to andrology clinical cases, between chatbots and andrologists, to assess the reliability of these technologies.

Material and method: We analyzed the responses of 32 experts, 18 residents and three chatbots (ChatGPT v3.5, v4 and Bard) to 25 andrology clinical cases. Responses were assessed on a Likert scale ranging from 0 to 2 for each question (0-false response or no response; 1-partially correct response, 2- correct response), on the basis of the latest national or, in the absence of such, international recommendations. We compared the averages obtained for all cases by the different groups.

Results: Experts obtained a higher mean score (m=11/12.4 σ=1.4) than ChatGPT v4 (m=10.7/12.4 σ=2.2, p = 0.6475), ChatGPT v3.5 (m=9.5/12.4 σ=2.1, p = 0.0062) and Bard (m=7.2/12.4 σ=3.3, p < 0.0001). Residents obtained a mean score (m=9.4/12.4 σ=1.7) higher than Bard (m=7.2/12.4 σ=3.3, p = 0.0053) but lower than ChatGPT v3.5 (m=9.5/12.4 σ=2.1, p = 0.8393) and v4 (m=10.7/12.4 σ=2.2, p = 0.0183) and experts (m=11.0/12.4 σ=1.4,p=0.0009). ChatGPT v4 performance (m=10.7 σ=2.2) was better than ChatGPT v3.5 (m=9.5, σ=2.1, p = 0.0476) and Bard performance (m=7.2 σ=3.3, p < 0.0001).

Conclusion: The use of chatbots in medicine could be relevant. More studies are needed to integrate them into clinical practice.

Level of evidence 4