As principles of competency-based medical education are implemented into graduate medical education [GME] programs, it is imperative that the assessments employed are reliable and valid. Internal structure is a core component of validity evidence that has been under studied. In this study, we examined elements of the internal structure for the American Board of Surgery In-Training Examination [ABSITE].

This national retrospective cohort study utilized performance data for general surgery residents from 2018 to 2023 to determine aspects of the internal structure for the ABSITE. The item difficulty, item discrimination, internal consistency, and dimensionality of the exam were calculated.

55,986 unique test results were obtained in the study. The ABSITE exhibits strong reliability, (Cronbach's alpha >0.9), and appropriate item difficulty (0.66) and item discrimination index (>0.2) for a formative examination. Results of the exploratory factor analysis reveals that the ABSITE is unidimensional.

The ABSITE is a unidimensional examination with strong reliability. Furthermore, the item difficulty and item discrimination levels of the exam are appropriate for a formative test. Future studies utilizing consensus methods could be employed to determine what specific construct is measured by the ABSITE.