More and more people are using large language models (LLMs) to seek out health information online. Although these tools have great potential to improve digital health literacy, not enough is known about their accuracy and consistency, especially in life-threatening conditions such as sepsis. The aim of this study was to test and compare the effectiveness of two popular LLMs, ChatGPT 4o and Gemini 2.5 Flash, in providing accurate and consistent answers to questions about sepsis.

A cross-sectional benchmarking study was conducted using a standardized set of sepsis-related questions, comprising two main categories: frequently asked questions (FAQs) and items drawn from the Surviving Sepsis Campaign (SSC) guidelines. The responses generated by the two models were independently assessed by two raters using the Global Quality Score (GQS), and reproducibility was evaluated by submitting each question twice.

Gemini significantly outperformed ChatGPT in overall quality and reproducibility. More specifically, 94% of Gemini’s responses received the highest GQS rating (GQS 5), compared to only 35.4% of the ChatGPT answers. Gemini also demonstrated higher reproducibility (97.5% vs. 76.5%). Both models underperformed in the “prevention” domain. Gemini showed greater potential than ChatGPT in delivering accurate and consistent sepsis-related health information, which is crucial for patients and caregivers alike.

These findings underscore the importance of rigorous benchmarking before integrating LLMs into digital health platforms, and illustrate a need for refinement of LLMs to enhance their reliability in public-facing health communication.