Despite its multifaceted importance, no validated or reliable tools assess the quality of the dictated operative note. This study determined the construct validity, interrater reliability, and internal consistency of a Structured Assessment Format for Evaluating Operative Reports (SAFE-OR) in general surgery.

SAFE-OR was developed by using consensus criteria set forth by the Canadian Association of General Surgeons. This instrument includes a structured assessment and a global quality rating scale. Residents divided into novice and experienced groups viewed and dictated a videotaped laparoscopic sigmoid colectomy. Blinded, independent faculty evaluators graded the transcribed reports using SAFE-OR.

Twenty-one residents participated in the study. Mean structured assessment scores (out of 44) were significantly lower for novice versus experienced residents (23.3 ± 5.2 vs 34.1 ± 6.0, t = .001). Mean global quality scores (out of 45) were similarly lower for novice residents (25.6 ± 4.7 vs 35.9 ± 7.6, t = .006). Interclass correlation coefficients were .98 (95% confidence interval, .96–.99) for structured assessment and .93 (95% confidence interval, .83–.97) for global quality scales. Cronbach ⍺ coefficients for internal consistency were .85 for structured assessment and .96 for global quality assessment scales.

SAFE-OR shows significant construct validity, excellent interrater reliability, and high internal consistency. This tool will allow educators to objectively evaluate the quality of trainee operative reports and provide a mechanism for implementing, monitoring, and refining curriculum for dictation skills.