The incorporation of artificial intelligence (AI) into the general surgery residency recruitment process holds great promise for overcoming limitations inherent to traditional application review methods. This study assesses the consistency of AI, particularly ChatGPT, in evaluating medical student performance evaluation (MSPE) letters in comparison to experienced human reviewers. While the results suggest that ChatGPT demonstrates greater consistency in grading than human reviewers, AI still has its limitations. This underscores the necessity for careful refinement and consideration in its implementation. While AI presents opportunities to enhance residency selection procedures, further research is imperative to fully grasp its capabilities and implications.