Implementation of the ACS/ APDS Resident Skills Curriculum reveals a need for rater training: An analysis using generalizability theory - 12/08/21
, Chalerm Eurboonyanun b, c , Yoon Soo Park d , Douglas Cassidy b , Sophia K. McKinley b , Isra Hamdi b , Roy Phitayakorn b , Emil Petrusa b , Denise W. Gee b, ⁎ Abstract |
Background |
The American College of Surgeons (ACS)/Association of Program Directors in Surgery (APDS) Resident Skills Curriculum includes validated task-specific checklists and global rating scales (GRS) for Objective Structured Assessment of Technical Skills (OSATS). However, it does not include instructions on use of these assessment tools. Since consistency of ratings is a key feature of assessment, we explored rater reliability for two skills.
Methods |
Surgical faculty assessed hand-sewn bowel and vascular anastomoses in real-time using the OSATS GRS. OSATS were video-taped and independently evaluated by a research resident and surgical attending. Rating consistency was estimated using intraclass correlation coefficients (ICC) and generalizability analysis.
Results |
Three-rater ICC coefficients across 24 videos ranged from 0.12 to 0.75. Generalizability reliability coefficients ranged from 0.55 to 0.8. Percent variance attributable to raters ranged from 2.7% to 32.1%. Pairwise agreement showed considerable inconsistency for both tasks.
Conclusions |
Variability of ratings for these two skills indicate the need for rater training to increase scoring agreement and decrease rater variability for technical skill assessments.
Le texte complet de cet article est disponible en PDF.Highlights |
• | Highlights for AJS article: Implementation of the ACS/APDS Resident Skills Curriculum Reveals a Need for Rater Training. |
• | Ratings of technical skill performances are not consistent with one another. |
• | Technical skill rating inconsistencies are partially attributable to the rater. |
• | Video-review ratings are more consistent than in-person technical skill assessments. |
• | Rating inconsistency is independent of resident vs. faculty rater status. |
Keywords : Rater training, Generalizability theory, Skill assessment
Plan
Vol 222 - N° 3
P. 541-548 - septembre 2021 Retour au numéro
Article précédent
- Predicting prospective resident entrustment: From evaluation to action
- Xiaodong (Phoenix) Chen, Amalia Cochran, Alan E. Harzman, E. Christopher Ellison
- An Institutional experience with primary hyperparathyroidism in the elderly over two decades
- Kayla O’Sullivan, Tina W.F. Yen, Kara Doffek, Sophie Dream, Ioanna Mazotas, Douglas B. Evans, Tracy S. Wang
Bienvenue sur EM-consulte, la référence des professionnels de santé.
L’accès au texte intégral de cet article nécessite un abonnement.
Déjà abonné à cette revue ?