Background

Residents deemed at risk for low performance on standardized examinations require focused attention and remediation.

Objective

To determine whether a remediation program for residents identified as at risk for failure on the Emergency Medicine (EM) Written Board Examination is associated with improved outcomes.

Intervention

All residents in 8 classes of an EM 1–3 program were assessed using the In-Training Examination. Residents enrolled in the Focused Board Intervention (FBI) remediation program based on an absolute score on the EM 3 examination of <70% or a score more than 1 SD below the national mean on the EM 1 or 2 examination. Individualized education plans (IEPs) were created for residents in the FBI program, combining self-study audio review lectures with short-answer examinations. The association between first-time pass rate for the American Board of Emergency Medicine (ABEM) Written Qualifying Examination (WQE) and completion of all IEPs was examined using the χ2 test.

Results

Of the 64 residents graduating and sitting for the ABEM examination between 2000 and 2008, 26 (41%) were eligible for the program. Of these, 10 (38%) residents were compliant and had a first-time pass rate of 100%. The control group (12 residents who matched criteria but graduated before the FBI program was in place and 4 who were enrolled but failed to complete the program) had a 44% pass rate (7 of 16), which was significantly lower (χ2  =  8.6, P  =  .003).

Conclusions

The probability of passing the ABEM WQE on the first attempt was improved through the completion of a structured IEP.

What was known

Remediation for inadequate medical knowledge requires a significant time commitment by faculty and residents, yet few studies have assessed the effectiveness of interventions.

What is new

Residents identified as at risk for failing the board examination through performance on the In-Training Examination were provided with individualized education plans that combined self-study review lectures with short-answer examinations.

Limitations

Small sample and single institution study may limit generalizability.

Bottom line

A focused board intervention significantly improved probability of passing the Emergency Medicine Board Examination for individuals with identified deficiencies.

Every incoming emergency medicine (EM) resident expects to finish his or her residency program with the skills and credentials of a competent EM physician. For many, part of proving this competency is to pass the American Board of Emergency Medicine (ABEM) examination. A key component of this examination is medical knowledge and the Accreditation Council for Graduate Medical Education (ACGME) has formally defined the basic skills for medical knowledge in all specialties. This competency states, “Residents must demonstrate knowledge about established and evolving biomedical, clinical, and cognate (eg, epidemiological and social-behavioral) sciences and the application of this knowledge to patient care. Residents are expected to: demonstrate an investigatory and analytic thinking approach to clinical situations and know and apply the basic and clinically supportive sciences which are appropriate to their discipline.”1 

The most common assessment of a resident is the objective standardized examination usually in the form of an In-Training Examination administered nationally. Criterion-referenced national examinations (eg, ABEM National In-Training Examination) are already being used in most residency programs across multiple specialties to identify strengths and weaknesses of a resident in certain core content areas.24 

According to the ABEM website, residents with higher In-Training Examination scores have a higher likelihood of passing the initial qualifying examination to be certified EM physicians than residents with lower examination scores.5 

Remediation for inadequate performance in this competency is not well-defined and typically requires a significant time commitment by the program director, faculty, and residents. Very little literature exists discussing the success and failure of remediation programs, and that which has been published focuses on improvements on in-service examinations themselves rather than on future success with the boards.68 We have developed and implemented a structured, individualized academic remediation plan based on performance on the In-Training Examination. This paper reviews the remediation program offered and how to determine whether the experience offered is associated with improved outcomes on the ABEM Written Qualifying Examination (WQE).

We performed a quasi-experimental, before-and-after study of residents in our 3-year EM training program who graduated residency between 2000 and 2008. All 64 residents who sat for the ABEM WQE were included in this study. Residents were identified as eligible for our remediation program, Focused Board Intervention (FBI), based on 1 of 2 criteria: absolute score on the EM 3 In-Training Examination of less than 70% or more than 1 SD from the national mean on the EM 1 or 2 In-Training Examination.

Residents enrolled in the FBI were given an individualized education plan (IEP) based on their areas of deficiency on the In-Training Examination. Modules corresponding to those areas were assigned from a program we designed with 19 units which cover the core curriculum of EM.

Each module consists of self-study audio review lectures (Emergency Medicine Review Course Audio program by Ohio State Chapter of the American College of Emergency Physicians) and a short-answer examination workbook (developed by and reviewed with the principal investigators). The residents were free to complete the examinations on their own schedule but with structured endpoints. Compliance with the program was defined by timely completion of all assigned modules prior to the next evaluation period. For the purpose of this study, the control group was defined as (1) historical controls, that is, residents who qualified for remediation prior to the development of the FBI program or (2) those who qualified for remediation but were resistant to the program (inadequate, incomplete, or untimely completion of required modules). The major outcome of this study was first-time pass rate on the WQE. The association between first-time pass rate for the ABEM WQE and compliance with IEPs was examined using the χ2 test for significance.

The Institutional Review Board provided exemption to this study.

In the 8 years between 2000 and 2008, 64 residents graduated from our residency program and sat for the ABEM WQE. Based on the defined inclusion criteria, 26 (41%) residents were eligible for enrollment in the FBI program. Of these, 10 (38%) were fully compliant with their IEPs and had a first-time pass rate of 100%. The control group, 12 historical residents and 4 residents who failed to successfully complete the FBI program, had a 44% pass rate (5 of 12 and 2 of 4, respectively). This difference was significant (χ2  =  8.6, P  =  .003). In addition, there were no significant differences in specialty board pass rate between those who completed the FBI and those who did not require remediation.

Three elements of the program accounted for its cost. The first entailed the purchase of the board review program ($505); the second element involved the time required to plan and design the fill-in form question sheets that accompany each module; and the third element was the maintenance of the program, including resident meetings and evaluations. The time commitment is estimated to be approximately 30 faculty hours for the planning phase for the entire program.

In this study, the probability of passing the ABEM WQE on the first attempt was improved in association with the completion of a structured IEP. Residents who were identified to be at risk for failure appear to have improved medical knowledge and test performance in association with this formalized remediation program.

Because the ACGME currently uses the pass rate of the national written and oral certifying examinations to assess the overall ability of a training program to educate their residents about medical knowledge, residency programs are left with surrogate markers, such as the In-Training Examination, that indicate success within training. Medical knowledge is a complex integration of data, facts, and pattern recognition within a clinical scenario. However, a remediation process must be specific and aimed at deficits documented for a particular individual, ie, an individualized educational plan. Our approach allows us to evaluate each member of the group and individualize their curriculum with a specific selection of modules based on their area(s) of weaknesses identified on the In-Training Examination. The workbook is designed to ensure that the resident learns certain basic facts from the audio portion which later serves as a review resource for board preparation. Adult learning is said to be most effective when the adult can proceed at his or her own pace.9 Allowing the resident to proceed with the remediation goals at his or her own pace (with some boundaries in the form of structured endpoints) while providing continuous feedback appears to be the most successful formula for enhancing medical knowledge.

The name choice for our intervention group was meant to lead the resident away from thinking involvement was punitive, but rather to see it as a form of feedback and an augmentation of their current learning process. By focusing on the deficiencies as seen in the percentage performance in categories on the In-Training Examination, we focus on the feedback nature of adult education. Feedback promotes learning and is central to the process of medical education.10 Residents expect some form of feedback after testing. Review of the In-Training Examination results provides the prime opportunity for feedback for the residents at risk for not passing the ABEM WQE. It has been suggested that adult learners welcome feedback, especially when it is based on their performance and tailored to their goals.11 We are attempting to motivate the resident to focus on deficient areas of knowledge with the goal of improving their chances of passing the ABEM WQE.

We based the use of the In-Training Examination as a useful form of feedback on research in other specialties of medicine. In a survey-based study, internal medicine residents alleged that the technique of In-Training Examination helped them identify areas for self-study and modify their study habits.12 Although not initially intended to be a practice test for the American Board of Radiology Written Board Examination, the In-Training Examination for Diagnostic Radiology was perceived as such by the radiology residents. The American College of Radiology Commission on Education and the Committee on Residency Training developed the examination with the purpose of providing residents with information to aid in the evaluation of their progress and providing program directors with data meant to help in the overall evaluation of the training program.13 In examining the correlation between scores on the 2 examinations, Baumgartner and Peterman13 found that the In-Training Examination score was a “strong predicator” of written board score. Furthermore, the researchers found that, “an in-training examination score of less than the 20th percentile identifies a resident at risk for poor performance or failure on the written board examination.” Our FBI enrollment criteria of 1 standard deviation below the national average for the resident's program year was arbitrary, but was developed with this study in mind based on a consensus of the attending physicians involved in implementing the FBI program and the available literature on identifying residents in need of remediation.14,15 Ideally, a statistically proven enrollment score would be a more powerful motivation for participation in the FBI.

The study has several limitations, including being conducted in a single program with a relatively small sample. An added potential limitation is the inability to account for programmatic changes in our residency curriculum over the study period. Annual evaluation of the residency curriculum leads to gradual changes in the didactic program and resident responsibilities. Over the 8 years of this study, our program has added and removed content to our curriculum, such as requirements for monthly online testing and a self-study program based on the resident's current rotation. Finally, the placement of residents in the FBI group based on a single In-Training Examination might be considered overly aggressive. A resident may not necessarily have deficiencies in medical knowledge but rather may have just had a bad test day. We have recently implemented a second, Internal Examination emulating the In-Training Examination. Although a senior resident may now be placed in the FBI based on either of these examinations, decisions regarding promotion and remediation are currently based on a composite of examination results. The Internal Examination is administered earlier in the resident year in order to obtain 2 data points for resident performance or an advanced assessment for graduating residents who may benefit from additional help.

When a program identifies a resident as falling behind in a basic competency, the implementation of a remedial education plan may correct this deficiency and help the resident find success.

Our focused board intervention program improved the probability of passing the ABEM WQE on the first attempt by having the resident complete an individualized structured learning program based on deficiencies identified through the in-training examination.

1
Accreditation Council for Graduate Medical Education
.
2011 Common Program Requirements
. .
2
Brateanu
A
,
Yu
C
,
Kattan
MW
,
Olender
J
,
Nielsen
C
.
A nomogram to predict the probability of passing the American Board of Internal Medicine examination
.
Med Educ Online
.
2012
;
17
:
18810
.
3
Johnson
GA
,
Bloom
JN
,
Szczotka-Flynn
L
,
Zauner
D
,
Tomsak
RL
.
A comparative study of resident performance on standardized training examinations and the American Board of Ophthalmology written examination
.
Ophthalmology
.
2010
;
117
(
12
):
2435
2439
.
4
Puscas
L
.
Otolaryngology resident in-service examination scores predict passage of the written board examination
.
Otolaryngol Head Neck Surg
.
2012
;
147
(
2
):
256
260
.
5
American Board of Emergency Medicine
.
2008 In-training examination overview
.
http://www.abem.org. Accessed June 24, 2009
.
6
Cheng
D
.
Board review course effect on resident in-training examination
.
Int J Emerg Med
.
2008
;
1
(
4
):
327
329
.
7
Kosir
MA
,
Fuller
L
,
Tyburski
J
,
Berant
L
,
Yu
M
.
The Kolb learning cycle in American Board of Surgery in-training exam remediation: the Accelerated Clinical Education in Surgery course
.
Am J Surg
.
2008
;
196
(
5
):
657
662
.
8
Harthun
NL
,
Schirmer
BD
,
Sanfey
H
.
Remediation of low ABSITE scores
.
Curr Surg
.
2005
;
62
(
5
):
539
542
.
9
Collins
J
.
Education techniques for lifelong learning: principles of adult learning
.
Radiographics
.
2004
;
24
(
5
):
1483
1489
.
10
Hewwon
MG
,
Little
ML
.
Giving feedback in medical education: verification of recommended techniques
.
J Gen Intern Med
.
1998
;
13
(
2
):
111
116
.
11
Knowles
MS
.
The Modern Practice of Adult Education
.
San Francisco, CA
.
Jossy-Bass Publishers
.
1980
.
12
Hawkins
RE
,
Sumption
KF
,
Gaglione
MM
,
Holmboe
ES
.
The in-training examination in internal medicine: resident perceptions and lack of correlation between resident scores and faculty predictions of resident performance
.
Am J Med
.
1999
;
106
(
2
):
206
210
.
13
Baumgartner
BR
,
Peterman
SB
.
Relationship between American College of Radiology in-training examination scores and American Board of Radiology written examination scores
.
Acad Radiol
.
1996
;
3
(
10
):
873
878
.
14
Ling
FW
,
Grosswald
SJ
,
Laube
DW
,
Carpentieri
AM
,
Crenshaw
C
.
The in-training examination in obstetrics and gynecology: an attempt to establish a remediation indicator
.
Am J Obstet Gynecol
.
1995
;
173
(
3
):
946
950
.
15
Bergen
PC
,
Littlefield
JH
,
O'Keefe
GE
,
Rege
RV
,
Anthony
TA
,
Kim
LT
,
et al.
Identification of high-risk residents
.
J Surg Res
.
2000
;
92
(
2
):
239
244
.

Author notes

All authors are with the Department of Emergency Medicine, New York Methodist Hospital. Annette Visconti, MD, is Associate Program Director; Theodore Gaeta, DO, MPH, is Vice Chair and Program Director; Michael Cabezon, MD, is Assistant Program Director; William Briggs, PhD, is Department Statistician; and Matthew Pyle, MD, is PGY-1 resident.

Presented at the Society for Academic Emergency Medicine Annual Meeting, New Orleans, LA, May 14–17, 2009.

Funding: The authors report no external funding sources for this study.