This study was undertaken to further evaluate the construct validity of the observed structured assessment of technical skills (OSATS) by comparing resident scores to faculty scores.

This study is a prospective blinded observational study. Four residents from each year (1-4) and 5 faculty members were examined. The OSATS examination was in the form of a dry laboratory with 10 stations: 4 laparoscopic and 6 open surgical skills. The sessions were videotaped and graded by the senior authors who used a task-specific checklist. The scoring of the videos was performed by the 2 senior authors blinded to the other examiners results. The examinations were videotaped, and the identity of the participants was blinded. The scores for each station were determined by adding all the numbers from the skills rating with a time score. Higher scores denote superior performance. Statistical analysis was performed with a nonparametric test (Kruskal-Wallis) for a total score with resident years divided into junior residents (first and second year), senior residents (third and fourth year), and faculty status as the independent variable. To determine the interrelater reliability between the 2 scores, the Kendall tau beta statistic was used.

The results show definite trends for 7 stations, with junior residents performing the worst and faculty performing the best. This trend was statistically significant for 6 of the 10 stations.

The OSATS examination has good construct validity that extends beyond residency to faculty.