Using an Accountability Program to Improve Psychiatry Resident Scores on In-Service Examinations

In-training examinations are used by most residency programs in the United States to measure residents' educational progress. These examinations are moderately predicted by trainees' performance on the United States Medical Licensing Examination (USMLE).^1–3  More important, performance on the in-training examination is a moderate-to-strong predictor of performance and pass rates on the American Board of Medical Specialties member board examinations,^4–9  giving residency program directors motivation to improve their residents' in-training examination (ITE) scores.

The majority of research on interventions to improve ITE performance has focused on residency programs' academic elements. The most frequently used strategy is adding a course to prepare trainees for the in-training or board-certifying examination. These courses have been shown to improve ITE scores,^10–15  although not universally, particularly with shorter courses.^16,17  Peer-led courses or study groups have been shown to work in neurology,¹⁸  but have had mixed results in a psychiatry^19,20  program. Academic half-days improved ITE scores in family medicine²¹  and internal medicine^22,23  programs, but not in psychiatry.²⁴  Duty hour restrictions have had no effect in either emergency medicine²⁵  or psychiatry²⁴  programs.

We could find little research in nonsurgical specialties regarding the effect of a consequences-based accountability program to improve ITE performance. In surgery, where ITE scores are used in promotion and fellowship decisions,²⁶  3 studies have assessed the impact of a mandatory remediation program,^27–29  with 1 showing statistically significant improvement.²⁹  One study in family medicine involved mandatory ITE remediation but showed no statistically significant score improvement.¹⁷ 

The objective of this study was to assess the effectiveness of a tiered, consequences-based accountability program in improving psychiatry residents' ITE scores.

Griffin Memorial Hospital is a public psychiatric hospital in central Oklahoma, with sister facilities for child and adult outpatient services and child inpatient services located on the same extended campus. We examined all psychiatry residents from Griffin Memorial Hospital's residency program between 2004 and 2013.

Between 2009 and 2011, residency program faculty began restructuring the program's didactics by (1) recruiting more board-certified faculty, (2) limiting rotation sites to those with academic faculty, (3) revising all course syllabi to include more structured teaching, (4) adding required scholarly projects, and (5) increasing residency responsibilities for teaching medical students. The restructuring produced no improvement in performance on the Psychiatry Resident-in-Training Examination (PRITE) scores, an annual 300-item test, with items divided unevenly between 2 subscales—psychiatry and neurology.³⁰  In response to this, the first iteration of the accountability program was put into place for the 2012 PRITE. Residents with scores at or below the 25th percentile were assigned 1 hour of study hall Friday morning before their usual duties, and were required to retake the examination.

For the 2013 PRITE, the second iteration of the accountability program was put into place. This version involved tiered consequences. All residents scoring below the 10th percentile were required to retake the examination and to have regular mandatory meetings with a mentor who assigned practice problems for a structured study hall. Residents scoring below the 30th percentile were required to retake the examination. To earn external moonlighting privileges, residents had to score above the 50th percentile or retake the examination if they did not. These consequences were applied cumulatively, such that a resident scoring at the fifth percentile, for example, would be subject to all consequences.

This study was approved by the Oklahoma Department of Mental Health and Substances Abuse Services Institutional Review Board.

The effectiveness of the accountability program was determined by residents' scores on the PRITE. Every resident received 4 scores for each subscale: (1) a raw score, reflecting the number of items answered correctly; (2) a score standardized on a distribution with mean and SD of 500 and 100, respectively; (3) percentile ranks for comparison to all other residents in the same training year; and (4) percentile ranks for comparison to all other residents. Using data from 2008 examinations, the internal consistency coefficients for the subscales were 0.90 for psychiatry and 0.61 for neurology.³⁰ 

Primary analysis of the PRITE data was conducted using a discontinuous growth curve model,³¹  in which score levels were allowed to vary before and after the accountability program was implemented. This model allowed for the separate estimation of the effect resulting from within-resident change over time and the effect from the accountability program. In a secondary model, 2 parameters were added to the model to test for differences between US and international medical graduates at baseline and after the accountability program was implemented. Parameters were estimated using full maximum likelihood estimation in the “nlme” package³²  in R version 3.0.3 (The R Foundation).

Secondary analyses included calculating 95% CIs for annual mean PRITE scores between 2011 and 2013. To assess different levels of resident aptitude, USMLE Step 1 or Comprehensive Osteopathic Medical Licensing Examination (COMLEX-USA) Level 1 scores were used to compare residents entering the program before or during 2010 and after 2010 using an independent t test. Each resident's score was standardized against the minimum passing score for that year's examination.

The study encompasses data from 67 residents with a mean age of 38 years old. Of these, 32 residents (48%) were women, and 49 (73%) were international medical graduates. The USMLE Step 1 or COMLEX-USA Level 1 scores showed that the scores of residents admitted to the program during or before 2010 were not statistically significantly different from those of residents admitted after 2010 (t[56] = 0.92; P = .36; d = 0.26).

The number of PRITE scores per resident varied, ranging from 1 to 4 scores per resident, with a mean and median of 2.8 and 3 scores per resident, respectively. Fifteen residents (22%) had at least 1 PRITE score before and after the implementation of the accountability program.

Results from the primary analysis of the standardized psychiatry subscale scores are found in table 1. The mean standardized psychiatry subscale score for first-year residents was 350.2 points, approximately 0.75 SDs below the national first-year average, and increased at a mean rate of 52.3 standardized points per year, approximately 0.50 SD annually. The accountability program was associated with an increase in psychiatry subscale scores by 51.1 standardized points. In terms of effect size, this is approximately a 0.50 SD change, a Cohen's d of 0.51, and the impact of an additional year of residency training. US medical graduates showed no difference in scores before and after the accountability program, compared to international medical graduates (P = .06).

We also compared the 95% CIs for the standardized psychiatry subscale scores for the years between 2011 and 2013. These years represent the period before any accountability program (2011), the first iteration of the accountability program (2012), and the second iteration of the program (2013). As shown in table 2, the CIs for 2011 (mean = 436.5) and 2012 (mean = 436.2) overlap entirely, and the mean CIs for 2011 and 2013 (mean = 537.5) are almost entirely different. The mean CIs for 2012 and 2013 are entirely distinct.

To determine if specific subgroups of residents differed in their improvement on the psychiatry subscale, we examined the 15 residents who had taken the PRITE at least once before and after the accountability program was implemented. There were no differences seen in these residents based on their first-year PRITE percentile scores or their postgraduate year status when the accountability program was implemented.

The mean standardized neurology subscale score for first-year residents was 459.4 points, approximately 0.10 SD below the mean for first-year residents, and increased at a mean rate of 26.9 standardized points annually. The effect of the accountability program was an increase in scores of 6.47 (P = .80), suggesting that the accountability program was not associated with an increase in performance on the neurology subscale.

Our results suggest that an accountability program improves ITE performance. The program may improve PRITE psychiatry subscale scores at a level comparable to that of an additional year of residency training. No effect was seen for the neurology subscale, potentially because of that subscale's lower reliability,³³  the residents' initially higher scores on that subscale, or another factor.

The accountability program was associated with improved PRITE scores, similar to the results for other interventions^15,19  in psychiatry programs. However, our study more closely maps onto a mandatory ITE remediation program in surgery.²⁹  Both studies involved more independent work on the part of residents, and had improvements in ITE performance. A potential advantage of the accountability program was that it permitted residents to study for the PRITE how, and for as long as, they chose. As such, different residents reported using different strategies shown to be successful in the literature, including reviewing questions from old examinations,¹⁵  studying with peers,¹⁹  or focusing on practice questions.²⁹  A variable strategy approach might perform better than a more singular and homogeneous intervention; however, future research is needed to explicitly test this.

Our study has limitations, including that it lacked a control group, which makes it difficult to rule out alternative causes for the observed effect. In addition, while the selection committee might have attempted to recruit better students, leading to a group of residents with higher PRITE scores in the absence of an accountability program, comparisons between older and newer residents found no differences between the groups in terms of USMLE Step 1 or COMLEX-USA Level 1 scores. Our study also involved only 1 residency program at 1 location, and the results may not necessarily be generalizable to other programs.

We will continue to use the accountability as it (1) has been associated with an increase in psychiatry ITE scores, (2) has been well received by faculty and most residents, and (3) has required almost no institutional resources. Resident scores have improved to the point that no residents have required mentoring. Future research should assess if the improvements in PRITE scores translate into improved clinical performance and improved performance on the board certification examination. This should take into consideration the relative effectiveness of different preparation methods among the residents.

After institution of an accountability program, scores on the PRITE's psychiatry subscale were 0.50 SD higher than before. The improvement on the psychiatry subscale associated with the accountability program was the equivalent of an extra year of residency training. Though low scorers showed more improvement in scores, an effect was also seen in low and high average scorers as well.