The development of health data warehouses (HDWs) in France raises concerns about personal data protection. Regulations require health research to use anonymized or pseudonymized data. This study evaluates a document pseudonymization tool developed by AP-HP, which integrates named entity recognition methods, within the NOVA Health cooperation group's project to establish an HDW at Limoges University Hospital.

This single-center observational study assessed the performance of the AP-HP pseudonymization algorithm on documents from Limoges University Hospital. The corpus included documents extracted from the Crossway clinical information system software. The study aimed to identify reidentifying labels revealing patient identity. Precision, recall, and F1-score were calculated by document type, hospital department, and patient.

The AP-HP algorithm achieved an overall F1-score of 79%, with performance varying from 15% to 96% among label types, document formats, and departments. Most false positives and negatives occurred in last names (25% and 21%), first names (23% and 18%), and dates (13% and 14%). The algorithm performed best for emails and phone numbers (0.5% and 0.5%; 4% and 3% for false positives and negatives, respectively). By document type, letters had the best performance (5% false positive, 6% false negative). Per patient, performance was good for up to six documents (80%) but dropped significantly for more than six (−25%).

While the AP-HP algorithm shows strong label identification, detailed error analysis and contextual adaptation of regular expressions at Limoges University Hospital are necessary to optimize accuracy and ensure full General Data Protection Regulation compliance in HDW data use.