This retrospective expert consensus study (PKUSSIRB No.202058145) aimed to establish expert consensus-derived evaluation criteria for orthodontic treatment outcomes using the Merge Ranking Method on post-treatment dental casts.

From patients treated at the Department of Orthodontics from January 2018 to December 2022, 216 cases were randomly selected for evaluation by 65 orthodontic experts using the Merge Ranking Method. Concurrently, nine objective indicators of the 216 post-treatment dental casts were measured by three researchers. The consistency analysis of experts’ subjective evaluation and researchers’ objective measurement was conducted, respectively. Through subjective-to-objective correlation analysis and regression analysis, the objective indicators significantly correlated with experts’ subjective evaluations were selected, their weights were determined, and the threshold values of grading evaluation were screened.

The 65 orthodontic experts demonstrated: (1) moderate pairwise consistency (mean Spearman's ρ = 0.560, 95% bootstrap CI: 0.556-0.564), (2) significant group-level concordance across two independent panels (Kendall's W = 0.544–0.606, all P < 0.001), and (3) near-perfect cross-panel reliability for 24 overlapping cases (Kendall's τ-b = 0.833–0.880, P < 0.001), confirming panel homogeneity for subsequent analyses. Inter-rater reliability among the three researchers showed excellent consistency (mean ICC = 0.835, 95% CI: 0.788–0.882, range: 0.736–0.920), paralleled by high intra-rater reliability (mean ICC = 0.832, 95% CI: 0.806–0.858, range: 0.715–0.948) across all 216 cases. Six objective indicators (occlusal relationship, overbite, alignment, overjet, occlusal contact, and buccal-lingual inclination) significantly predicted expert evaluations in a regression model (cumulative R ² = 0.598, P < 0.001). The threshold values for grading orthodontic treatment outcomes as Excellent, Good, Fair, Poor, and Worst were screened to be 1.846, 2.454, 3.492, and 4.312, respectively.

This expert consensus study demonstrated moderate consistency in subjective orthodontic outcome evaluation, with the occlusal relationship emerging as the primary quality determinant. The developed Merge Ranking Method addressed conventional ranking limitations through its innovative two-stage approach: initial segmented evaluation reduced expert fatigue, while subsequent dynamic adjustments improved borderline case classification.