Context.—Factors that correlate with success or failure on the American Board of Pathology (ABP) examination are not known. Other medical residency programs have shown that standardized test scores correlate with specialty board examination scores; however, data from pathology programs are lacking.
Objective.—To investigate whether the 2-digit score on step 1 of the United States Medical Licensing Examination (USMLE) was correlated with ABP examination performance at a large university pathology program.
Design.—Nine years of data (2001–2009) from pathology residents (n = 72) at the University of Pittsburgh Medical Center (UPMC, Pittsburgh, Pennsylvania) was collected from existing files and deidentified. Step 1 USMLE 2-digit scores and ABP failure rates for first-time test takers were compared. Results are reported as the percentage of residents who failed either the anatomic pathology or clinical pathology part of the ABP examination in cohorts by their USMLE 2-digit score (≤80, 81–85, 86–89, ≥90).
Results.—The rolling 5-year (2005–2009) ABP average failure rate for first-time test takers of the anatomic pathology examination was 3.1% (UPMC) and 14.1% (nationally); in clinical pathology, it was 13.8% (UPMC) and 23.6% (nationally). At UPMC, no resident failed the anatomic pathology or clinical pathology parts of the ABP examination if his or her 2-digit USMLE step 1 score was 90 or more across 9 years of training (2001–2009).
Conclusions.—In the UPMC pathology program, 2-digit scores on USMLE step 1 of 90 or more and 80 or less were strong measures of ABP first-time pass/failure rates, whereas scores of 81 to 89 were less-accurate measures. The USMLE step 1 score is one of many criteria that can be used for screening applicants for a pathology residency program.
Selection of medical students to residency programs includes a variety of factors, including clinical rotation performance, medical school grades, letters of recommendation, interviews, and evaluation of standardized test scores. Step 1 of the United States Medical Licensing Examination (USMLE), completed during medical school, is an objective tool for screening and comparing residency applicants across a wide range of medical training programs. Since the mid-1990s, the USMLE has largely replaced the National Board of Medical Examiners test. Previous studies have shown a relationship between results on part 1 of the National Board of Medical Examiners test and future performance on subspecialty board examinations.1,2 More recently, a similar relationship has been seen with USMLE step 1 scores and board performance in medical residencies3–5 and dental medicine training programs.6 Some training programs, such as orthopedic surgery programs, have shown less-significant correlations between step 1 scores and pass/failure rates on future subspecialty boards.7–9
For editorial comment, see p 1244.
Selection of pathology residents is a complex task; because many medical school programs do not offer a pathology clerkship, significant variation exists in the level of preresidency pathology training and experience. To that end, the USMLE step 1 score provides one objective tool for screening applicants and thus possibly helping to risk-stratify applicants' future success in their subsequent standardized test-taking ability, including the examination for American Board of Pathology (ABP) certification.
One measure of individual and program training success is passing the ABP examination. The University of Pittsburgh Medical Center (UPMC, Pittsburgh, Pennsylvania) pathology department's rolling-pass average on the ABP examination is consistently better than the national average, but factors that correlate with success or failure for first-time test takers are unknown. The UPMC pathology residency program is approved by the Accreditation Council for Graduate Medical Education, and the training is based on a Center of Excellence model in which residents rotate through anatomic pathology and clinical pathology subspecialties. To our knowledge, there are no studies in the literature specifically looking at USMLE step 1 results as indicators of board success in pathology. Other residency programs, including pediatrics, internal medicine, and radiology, have previously shown that step 1 examination scores in medical school correlate with subspecialty board scores.3–5 The aim of this study was to investigate whether the 2-digit scores on step 1 of the USMLE correlate with ABP performance in the UPMC pathology program.
MATERIALS AND METHODS
Subject and Program Descriptions
The residency program at the UPMC recruits single-track (anatomic pathology [AP], AP/neuropathology, or clinical pathology [CP]) and combined track (AP/CP) residents. The step 1 score on the USMLE is used as an initial application filter, before screening of applications and interview invitation, and in subsequent placement on the match list. The training program is a Center of Excellence multiple-site program, which includes CP training that is based on organ systems and subspecialties.
During each anatomic and clinical pathology rotation, residents work closely with a strong group of pathology assistants and, in many instances, in-training pathology fellows. The fellowship program has 15 subspecialty fellowships with 22 fellows. In addition to our core and elective rotations, there are 4 h/wk of mandatory didactic teaching conferences. Starting in the 2007–2008 academic year, a CP didactic curriculum program was started, which included laboratory management didactic sessions.
Nine years of data (2001–2009) from the 72 UPMC pathology residents was collected from existing files and deidentified. Five residents with incomplete data were excluded from analysis, leaving 93.1% of the resident sample (n = 67), which included 2 residents with National Board of Medical Examiners test scores. Their National Board of Medical Examiners 2-digit test scores were used as a surrogate for the USMLE 2-digit score.
Data Analysis
Two-digit scores from step 1 of the USMLE and the ABP failure rates for first-time test takers were compared. Results are reported as the percentage of residents who failed any part of the ABP examination in cohorts by their USMLE 2-digit scores (≤80, 81–85, 86–89, ≥90) and by resident groups (all); AP/CP, anatomic pathology/clinical pathology dual-track residents; international medical graduates (graduates from medical schools outside the United States); US medical graduates (graduates of medical schools in the United States); medical school graduates with dual postgraduate degrees (eg, MD/PhD); and medical graduates with a single doctor of medicine degree. Categorical variables were analyzed using the Fisher exact and χ2 tests and 3 discrete, 2-digit USMLE step-1 score echelons: 80 or less, 81 to 89, and 90 or better. Statistical significance was defined as P < .05.
RESULTS
The 5-year (2005–2009), nationally reported, ABP failure rate for first-time takers of the AP part of the test was lower for residents in the UPMC pathology program compared with the national average: 3.1% at UPMC versus 14.1% nationally (Table). The first-time failure rate for the CP part of the test was also lower for UPMC residents than the national average: 13.8% at UPMC versus 23.6% nationally (Table).
Across 9 years of training (2001–2009) at UPMC, no resident failed the AP or CP part of ABP examination (first-time test takers) if his or her 2-digit USMLE step-1 score was 90 or better (Figure). There was no statistically significant difference in ABP examination failure rates between residents scoring 80 or less versus those that scored 81 to 89 on step 1 of the USMLE for any resident group (P > .05). However, statistically significant differences in failure rates were found between residents scoring 80 or less and those scoring 90 or better (P = .002) as well as between those scoring 81 to 89 and those scoring 90 or better (P = .02).
American Board of Pathology (ABP) failure rates as a percentage of different resident groups in cohorts by their United States Medical Licensing Examination (USMLE) step 1 scores. Abbreviations: AP/CP, anatomic pathology/clinical pathology dual-track residents; IMG, International medical graduates (graduates from medical schools outside the United States); US grad, US medical graduates (graduates from medical schools in the United States); MD, medical school graduates with a single degree; MD/PhD, medical school graduates with dual degrees.
American Board of Pathology (ABP) failure rates as a percentage of different resident groups in cohorts by their United States Medical Licensing Examination (USMLE) step 1 scores. Abbreviations: AP/CP, anatomic pathology/clinical pathology dual-track residents; IMG, International medical graduates (graduates from medical schools outside the United States); US grad, US medical graduates (graduates from medical schools in the United States); MD, medical school graduates with a single degree; MD/PhD, medical school graduates with dual degrees.
The overall failure rate in 2001–2009 was 20.9% (14 of 67) for all first-time ABP test takers at UPMC (AP, CP, AP/CP, and AP/neuropathology tracks) and 24.1% (13 of 54) for AP/CP residents alone. Among all first-time test takers with a USMLE step 1 score of 80 or less, the overall failure rate was 43% (6 of 14). All (100%) of US medical graduates at UPMC with a USMLE step 1 score of 80 or less (n = 3) failed either the AP, CP, or both components (all were AP/CP track) compared with 27% (3 of 11) of the international medical graduates with a 2-digit score of 80 or less. The overall failure rate for US medical graduates at UPMC was 27% (8 of 30), which was greater than for international medical graduates at UPMC (16%; 6 of 37). Residents with dual postgraduate degrees had a similar overall failure rate (20%; 4 of 20) compared with residents with a single doctor of medicine degree (21%; 10 of 47).
CONCLUSIONS
The ability to pass the examination for ABP certification is an objective indicator that allows a comparison across a multitude of pathology residency programs. The results of this small study show that residents whose 2-digit score is 90 or more on step 1 of the USMLE do not fail any part of their ABP examination, whereas there appears to be an increased risk of failing the ABP test with a 2-digit score less than 80. A number of possible confounding variables could affect the significance of our results. Our results should not suggest that standardized scores can be used as the sole criterion for predicting a future applicant's ability to be an excellent pathologist. Such scores are only measures of medical knowledge heavily admixed with test-taking finesse. However, with the task of screening hundreds of residency applications, these results have given us one objective measure to filter candidate applications. Although tempting to speculate that these results may also allow a program to predict success in other standardized tests, that is beyond the scope of this article, which included only a comparison of results from step 1 of the USMLE with ABP examination results.
Although there are other methods to assess competency, the ABP examination provides one objective, national measure of medical knowledge, which indicates a graduating resident has an adequate level of knowledge to practice pathology. Although standardized test scores are but one means of assessing medical knowledge, they are used as a measure of the quality of a training program. They are one indicator of a resident's education, which is of value to future applicants and accreditation committees who review such program data. Most agree that the USMLE step 1 score should not be the sole criterion for selecting residents,10 but it is one objective measure that can help program committees screen applicants and can alert program administrators to those current residents at greatest risk of failing the APB examination.
In the UPMC pathology program, when residents scored 90 or more on step 1 of the USMLE, there was a 0% ABP failure rate for first-time test takers. The greatest rate of ABP failure for first-time test takers was for those whose 2-digit USMLE step 1 score was 80 or less, with greater failure rates for US medical graduates. There were statistically significant differences in ABP examination failure rates between those residents who scored less than 90 and those who scored 90 or more on step 1 of the USMLE. There is evidence from nonpathology residency programs that strong, direct correlations exist between board performance and subsequent clinical performance among those residents at the upper or lower extremes of standardized test-scoring echelons.11
There was a slightly lower failure rate for those residents with step 1 USMLE 2-digit scores of 81 to 89; however, US graduates at UPMC had the greatest failure rates (35.7%; 5 of 14) compared with other resident groups, especially international medical graduates (16.7%; 3 of 18). In our pathology program, 24% of the international medical graduates had at least 2 years of previous pathology training, which is likely a confounding factor that partially explains the lower ABP failure rates for residents with scores of 81 to 89. Additional research training did not appear to correlate with ABP examination success because there was a similar ABP failure rate for those with MD/PhD training compared with MD-only training.
Brener et al12 suggest that although USMLE scores can be used as a selection criterion, individual program committees are best suited for determining minimum score cutoffs based on program-specific research data from their own institution. Based on the results of this study, one of the current applicant-screening criteria at our institution includes no USMLE failures and a minimum 2-digit score of 80 for step 1 of the USMLE. In addition, we prefer candidates who have a 2-digit USMLE step score of 90 or more. For international medical graduates, we prefer applicants with 2 or more years of pathology-residency training experience.
In the literature, some studies have suggested that standardized tests taken closer to residency training,2 rather than earlier in medical school, better correlate with success on specialty-board performance. However, all of these studies are based on clinically oriented specialties (dermatology, orthopedics, and preventive medicine), and clinical medicine (management, treatment, and therapies) is tested more heavily in USMLE steps 2 and 3 than in the USMLE step 1. Although steps 2 and 3 may also predict success, we focused only on step 1 scores in this study, and thus, we cannot make conclusions about the other standardized examination scores.
There is value in assessing whether in-service pathology examinations have a correlation with board success. Two orthopedic surgery programs and one anesthesiology program showed that their in-service examinations were a better predictor of board success than the USMLE scores;7–9 nevertheless, a USMLE step 1 score greater than the national mean continued to be a significant predictor in one of the studies.8 All US pathology programs participate in the American Society of Clinical Pathology's Resident In-Service Examination (RISE),13 an annual multiple-choice, timed test that assesses anatomic and clinical pathology knowledge, including molecular pathology and laboratory management. This is not administered as a proctored examination in all programs, so it is difficult to make national RISE score comparisons. Recent communication by the American Society of Clinical Pathology indicates that there is a strong correlation between RISE scores in AP for postgraduate year-4 residents and the pass rate on the AP portion of the ABP examination from a collaborative study done in 2006.13 Our program does administer the RISE as a proctored examination, and a comparison of RISE scores with ABP first-time takers is a possible future program project. However, any RISE score correlation would serve as an in-training guide because it is not available as an applicant-screening tool. For measuring overall pass-rate success on resident boards, other studies have shown that the combination of in-service training tests and USMLE step 1 scores is the best predictor of those residents who were likely to pass their board examinations.14 It would be worthwhile to compare both USMLE and pathology RISE scores to determine whether low RISE scores could further stratify a pathology resident at risk for board failure and thus in need of a more focused individual learning plan.
Despite 9 years of data, a major limitation to this study is the small number of residents (n = 67), with data collection beginning in 2001. The USMLE data were missing for 5 applicants, one of whom failed the CP part of the ABP examination. We did not control for multiple confounding factors (eg, previous pathology training/knowledge, English as a first language, length of time between graduating from medical school and starting residency, and time spent or type of study program for the board examination). In addition, we did not control for medical school, medical school grades, class rank, level of prior pathology training, and letters of recommendation, none of which can be accurately standardized. Our program training has been consistent during the past 9 years, with a change in program director in 2006 (due to relocation, not program performance). The most significant changes to the program during the past 3 years was the implementation of a weekly CP didactic curriculum, which includes 6 lectures focused on laboratory management, and a more rigorous electives program with an expanded menu of options.
In conclusion, in the UPMC pathology residency program for the years 2001–2009, the 2-digit USMLE step 1 scores of 90 or more and 80 or less correlated with the pass/fail rate of our first-time ABP test takers. These results suggest the 2-digit USMLE step 1 score is now an important screening tool in selecting candidates to interview for our program.
Special thanks to Trish Zrimsek, the academic manager of the UPMC pathology residency program, and Milon Amin, MD, a UPMC anatomic pathology/clinical pathology resident, for their help in this project.