Focused Board Intervention (FBI): A Remediation Program for Written Board Preparation and the Medical Knowledge Core Competency

Every incoming emergency medicine (EM) resident expects to finish his or her residency program with the skills and credentials of a competent EM physician. For many, part of proving this competency is to pass the American Board of Emergency Medicine (ABEM) examination. A key component of this examination is medical knowledge and the Accreditation Council for Graduate Medical Education (ACGME) has formally defined the basic skills for medical knowledge in all specialties. This competency states, “Residents must demonstrate knowledge about established and evolving biomedical, clinical, and cognate (eg, epidemiological and social-behavioral) sciences and the application of this knowledge to patient care. Residents are expected to: demonstrate an investigatory and analytic thinking approach to clinical situations and know and apply the basic and clinically supportive sciences which are appropriate to their discipline.”1 

The most common assessment of a resident is the objective standardized examination usually in the form of an In-Training Examination administered nationally. Criterion-referenced national examinations (eg, ABEM National In-Training Examination) are already being used in most residency programs across multiple specialties to identify strengths and weaknesses of a resident in certain core content areas.2–4 

According to the ABEM website, residents with higher In-Training Examination scores have a higher likelihood of passing the initial qualifying examination to be certified EM physicians than residents with lower examination scores.5 

Remediation for inadequate performance in this competency is not well-defined and typically requires a significant time commitment by the program director, faculty, and residents. Very little literature exists discussing the success and failure of remediation programs, and that which has been published focuses on improvements on in-service examinations themselves rather than on future success with the boards.6–8 We have developed and implemented a structured, individualized academic remediation plan based on performance on the In-Training Examination. This paper reviews the remediation program offered and how to determine whether the experience offered is associated with improved outcomes on the ABEM Written Qualifying Examination (WQE).

We performed a quasi-experimental, before-and-after study of residents in our 3-year EM training program who graduated residency between 2000 and 2008. All 64 residents who sat for the ABEM WQE were included in this study. Residents were identified as eligible for our remediation program, Focused Board Intervention (FBI), based on 1 of 2 criteria: absolute score on the EM 3 In-Training Examination of less than 70% or more than 1 SD from the national mean on the EM 1 or 2 In-Training Examination.

Residents enrolled in the FBI were given an individualized education plan (IEP) based on their areas of deficiency on the In-Training Examination. Modules corresponding to those areas were assigned from a program we designed with 19 units which cover the core curriculum of EM.

Each module consists of self-study audio review lectures (Emergency Medicine Review Course Audio program by Ohio State Chapter of the American College of Emergency Physicians) and a short-answer examination workbook (developed by and reviewed with the principal investigators). The residents were free to complete the examinations on their own schedule but with structured endpoints. Compliance with the program was defined by timely completion of all assigned modules prior to the next evaluation period. For the purpose of this study, the control group was defined as (1) historical controls, that is, residents who qualified for remediation prior to the development of the FBI program or (2) those who qualified for remediation but were resistant to the program (inadequate, incomplete, or untimely completion of required modules). The major outcome of this study was first-time pass rate on the WQE. The association between first-time pass rate for the ABEM WQE and compliance with IEPs was examined using the χ² test for significance.

The Institutional Review Board provided exemption to this study.

In the 8 years between 2000 and 2008, 64 residents graduated from our residency program and sat for the ABEM WQE. Based on the defined inclusion criteria, 26 (41%) residents were eligible for enrollment in the FBI program. Of these, 10 (38%) were fully compliant with their IEPs and had a first-time pass rate of 100%. The control group, 12 historical residents and 4 residents who failed to successfully complete the FBI program, had a 44% pass rate (5 of 12 and 2 of 4, respectively). This difference was significant (χ²  =  8.6, P  =  .003). In addition, there were no significant differences in specialty board pass rate between those who completed the FBI and those who did not require remediation.

Three elements of the program accounted for its cost. The first entailed the purchase of the board review program ($505); the second element involved the time required to plan and design the fill-in form question sheets that accompany each module; and the third element was the maintenance of the program, including resident meetings and evaluations. The time commitment is estimated to be approximately 30 faculty hours for the planning phase for the entire program.

In this study, the probability of passing the ABEM WQE on the first attempt was improved in association with the completion of a structured IEP. Residents who were identified to be at risk for failure appear to have improved medical knowledge and test performance in association with this formalized remediation program.

Because the ACGME currently uses the pass rate of the national written and oral certifying examinations to assess the overall ability of a training program to educate their residents about medical knowledge, residency programs are left with surrogate markers, such as the In-Training Examination, that indicate success within training. Medical knowledge is a complex integration of data, facts, and pattern recognition within a clinical scenario. However, a remediation process must be specific and aimed at deficits documented for a particular individual, ie, an individualized educational plan. Our approach allows us to evaluate each member of the group and individualize their curriculum with a specific selection of modules based on their area(s) of weaknesses identified on the In-Training Examination. The workbook is designed to ensure that the resident learns certain basic facts from the audio portion which later serves as a review resource for board preparation. Adult learning is said to be most effective when the adult can proceed at his or her own pace.9 Allowing the resident to proceed with the remediation goals at his or her own pace (with some boundaries in the form of structured endpoints) while providing continuous feedback appears to be the most successful formula for enhancing medical knowledge.

The name choice for our intervention group was meant to lead the resident away from thinking involvement was punitive, but rather to see it as a form of feedback and an augmentation of their current learning process. By focusing on the deficiencies as seen in the percentage performance in categories on the In-Training Examination, we focus on the feedback nature of adult education. Feedback promotes learning and is central to the process of medical education.10 Residents expect some form of feedback after testing. Review of the In-Training Examination results provides the prime opportunity for feedback for the residents at risk for not passing the ABEM WQE. It has been suggested that adult learners welcome feedback, especially when it is based on their performance and tailored to their goals.11 We are attempting to motivate the resident to focus on deficient areas of knowledge with the goal of improving their chances of passing the ABEM WQE.

We based the use of the In-Training Examination as a useful form of feedback on research in other specialties of medicine. In a survey-based study, internal medicine residents alleged that the technique of In-Training Examination helped them identify areas for self-study and modify their study habits.12 Although not initially intended to be a practice test for the American Board of Radiology Written Board Examination, the In-Training Examination for Diagnostic Radiology was perceived as such by the radiology residents. The American College of Radiology Commission on Education and the Committee on Residency Training developed the examination with the purpose of providing residents with information to aid in the evaluation of their progress and providing program directors with data meant to help in the overall evaluation of the training program.13 In examining the correlation between scores on the 2 examinations, Baumgartner and Peterman13 found that the In-Training Examination score was a “strong predicator” of written board score. Furthermore, the researchers found that, “an in-training examination score of less than the 20th percentile identifies a resident at risk for poor performance or failure on the written board examination.” Our FBI enrollment criteria of 1 standard deviation below the national average for the resident's program year was arbitrary, but was developed with this study in mind based on a consensus of the attending physicians involved in implementing the FBI program and the available literature on identifying residents in need of remediation.14,15 Ideally, a statistically proven enrollment score would be a more powerful motivation for participation in the FBI.

The study has several limitations, including being conducted in a single program with a relatively small sample. An added potential limitation is the inability to account for programmatic changes in our residency curriculum over the study period. Annual evaluation of the residency curriculum leads to gradual changes in the didactic program and resident responsibilities. Over the 8 years of this study, our program has added and removed content to our curriculum, such as requirements for monthly online testing and a self-study program based on the resident's current rotation. Finally, the placement of residents in the FBI group based on a single In-Training Examination might be considered overly aggressive. A resident may not necessarily have deficiencies in medical knowledge but rather may have just had a bad test day. We have recently implemented a second, Internal Examination emulating the In-Training Examination. Although a senior resident may now be placed in the FBI based on either of these examinations, decisions regarding promotion and remediation are currently based on a composite of examination results. The Internal Examination is administered earlier in the resident year in order to obtain 2 data points for resident performance or an advanced assessment for graduating residents who may benefit from additional help.

When a program identifies a resident as falling behind in a basic competency, the implementation of a remedial education plan may correct this deficiency and help the resident find success.

Our focused board intervention program improved the probability of passing the ABEM WQE on the first attempt by having the resident complete an individualized structured learning program based on deficiencies identified through the in-training examination.