We read with interest the article, “A Critical Disconnect: Residency Selection Factors Lack Correlation With Intern Performance,” by Burkhardt et al and would like to commend the authors for their valuable investigation into the status quo of the residency match process.1 Their conclusion corroborates our experience. All residency programs seek to identify and recruit whom they believe are the most promising medical school graduates—typically based on USMLE scores, class ranking, clinical grades, letters of recommendation, etc.2 The assumption in this ranking process is that competitiveness via these criteria is the best available metric for judging future performance of resident physicians. Existing evidence suggests that programs are not particularly efficient at determining whether applicants selected in this manner will become top performers during their residency training.3
We suspected that current applicant metrics are not adequate predictors and ultimately do not correlate strongly with eventual resident performance. We examined National Resident Matching Program (NRMP) rank order list data over a 15-year period of anesthesiology residents from our institution, with class sizes varying from 6 to 12 residents per year. We were able to ask 4 longstanding, full-time faculty members who were present for the entire residency experience of these classes to reflect and “re-rank” the matched candidates relative to each other. We were limited to faculty who felt that they remembered the candidates well enough to complete the task. Each eligible faculty member independently ranked the past graduates of each class from highest to lowest compared to their class peers. The ranking was based on the faculty's recollection of the residents after working with them during their residency. Faculty essentially re-ranked these residents based on their overall impression of the graduates after they experienced their individual strengths, shortcomings, and limitations. Descriptive analysis was performed comparing correlation between the percentile in the actual NRMP rank list position vs faculty raters' ranking of each resident.
From our preliminary findings, there was general agreement among the faculty raters with considerable correlation (R = 0.567). While some variation in rankings among the faculty existed, no faculty member was an obvious outlier. Despite some evidence that residents taken higher in the NRMP rank list received higher impression ratings from the faculty, the association was quite weak with a lower correlation (R = 0.204).
Our findings reinforce the authors' results and suggest some insight into how difficult it is to identify and predict potential for clinical excellence. Extensive time and resources are devoted to parameters that, in our opinion, weakly predicts future resident performance. Exploration of nontraditional metrics may prove more valuable than currently emphasized parameters.
