To identify the current formats of standardized letters of recommendation (SLORs) and evaluate their characteristics, the distribution of applicants’ ratings, correlation between SLOR domain ratings and conventional application metrics, and potential biases.

We evaluated all applications submitted to our residency program for the 2020-2021 urology match. Two main formats of SLOR were identified. We extracted application characteristics and SLOR domain ratings.

Ninety SLORs from 82 applicants were reviewed. Applicants were highly rated among top tiers in both formats. Some correlations were observed between domain ratings and application metrics such as Step 1 and Step 2 Clinical Knowledge scores, and percentage of Honors in core clinical clerkships. No statistically significant differences were found between female and male applicants in terms of domain ratings. Alpha Omega Alpha members received higher ratings in “urology resident potential,” “academic urologist potential,” and “performance as a sub-intern” domains. Applicants from top 40 US medical schools performed better as sub-interns, and were more likely to be ranked higher. Letters from home institutions were associated with higher ratings in several domains. In-person vs virtual interactions received similar ratings except for “communication”.

While it is promising to observe such number of SLORs submitted for the first time in urology, the current formats could benefit from further refinement in their structures and domains to distinguish between highly qualified urology applicants more efficiently. Given the transition in Step 1 score reporting to pass/fail outcome, the need for a reliable urology-specific SLOR will be critical.