The main objective was to prove the robustness of the modified Lund-Kennedy staging system and its use in the clinical research group. Secondary objectives were to evaluate the physicians’ homogeneity, identify outliers with an unacceptable agreement and define factors for questionable agreement within the group of raters.

Anonymized endoscopic photos of patients with chronic rhinosinusitis were assessed by independent raters from a clinical research group. The level of agreement between raters was calculated using intra-class correlation and weighted kappa coefficient. Clusters of similarity were identified using Inter-Item Correlation Matrix. The weighted kappa coefficient was calculated for the most homogeneous group and outliers. Age, sex, consultancy years, combined clinical and research work assessed by 5 senior peers were also statistically compared between raters.

Intraclass-correlation coefficients were 0.75 and 0.95 for respectively single and average measures. Single measures value for most homogenous raters was 0.97 (weighted kappa 0.88, (P<0.001). One outlier with less research work score had an unacceptable agreement for single measures coefficient values with the 2 most homogenous raters (respectively 0.59, weighted kappa 0.15, P=0.32 and 0.57, weighted kappa 0.197, P=0.32). Pooled groups were similar in age (P=0.3), sex (P=0.1) and consultancy years (P=0.2) but significantly differentiated in peer-assessed clinical and research work score (P<0.001).

Even with a perfect overall agreement, careful examination of correlation matrix revealed an obvious outlier with less than ideal performance. The method may be helpful when studies using endoscopic staging system are designed to involve researchers from different backgrounds. When exploring the most common factors, education and clinical experience play a paramount role.