To investigate whether large language models (LLMs) can assist with maternal and neonatal health care in low- and middle-income countries.

We evaluated the ability of GPT-4o to generate accurate answers across countries in 4 domains related to maternal and neonatal health: (1) prevalence of conditions when generating medical case examples; (2) prevalence of conditions in countries without reliable national prevalence data; (3) standardized medical examination questions; and (4) subjective health-related questions. We used the GPT-4o Application Programming Interface except for domain 2, for which we used ChatGPT, and used repeated prompts to guarantee statistical significance of answers. We utilized publicly available data from 6 WHO regions and 204 countries.

We observed challenges for LLMs to provide accurate answers on a global scale. Medical cases generated by GPT-4o did not reflect true prevalences of outcomes, over-representing the Americas. GPT-4o demonstrated explicit bias, giving lower rankings for subjective health-related topics to countries with high infant mortality rates. In 44% of cases, GPT-4o provided pregnancy-related statistics in regions where those statistics were not available, while not acknowledging the uncertainty, and nearly half (46.7%) of the source citations were erroneous. GPT-4o answered the majority (79%) of pregnancy-related medical examination questions correctly but made errors when answering based on prevalent health issues in specific regions while overlooking symptoms.

Identified challenges in using GPT-4o highlight important limitations in applying general-purpose LLMs to guide maternal and neonatal healthcare in low- and middle-income countries. These findings can guide further studies and solutions in fine-tuning LLMs on contextualized data.