To the Editor.—We read with great interest the editorial “Recent Advances in Anatomic and Clinical Pathology Board Examination Pass Rates” by Brian D. Adkins, MD.1 Historic board pass rates are nicely summarized, and the author speculates on the many reasons for an upward trend in pass rates, although this was only significant when looking at all test takers, but not when breaking out first time and repeat test takers.
We believe that training programs began to place greater emphasis on board pass rates when the Accreditation Council for Graduate Medical Education made board pass rates an outcome for continued accreditation of a training program. We agree there is an increasing number and better quality of tools for board preparation.
Readers may be interested to know that the American Board of Pathology (ABPath) uses criterion referenced tests (CRTs), and not norm-referenced tests (NRTs) for its certification exams. CRTs are intended to measure how well a person has learned a specific body of knowledge and skills. On a CRT, the passing or cut score is determined in advance by a committee of subject matter experts (SMEs) in the field. The candidate's performance (ie, mastery of the subject matter) is compared with the cut score and not to other test takers. Theoretically, all candidates could pass, or all could fail depending on how they performed relative to the passing score. A driver's test is an example of CRTs, because the goal is to see whether the test taker is skilled enough to be granted a driver's license, not to see if one test taker is more skilled than another test taker. In contrast, NRTs compare test takers with each other. NRTs are designed to sort and rank candidates “on the curve,” not to see if they met a standard or criterion. NRTs, by design, always have a failure rate.
The ABPath uses the modified Angoff method, the most commonly used and psychometrically valid method for determining the pass point for an exam. Using this method, a group of SMEs are instructed to think of a minimally qualified candidate, because we want our exam to separate candidates who should be certified from those who should not be certified. The SMEs consider each item on a test and estimate the percentage of minimally qualified candidates who would answer the item correctly. This estimates the difficulty (Angoff rating) of each item. If the SMEs estimate that 90% of minimally qualified candidates would get the item correct, it is an easy question, but if they estimate that only 20% of minimally qualified candidates would get it correct, it is a difficult question. The average or sum of the Angoff ratings is used to determine the provisional cut score (ie, the score that a borderline candidate, someone who should just barely pass) would be expected to achieve. That provisional cut score is retrospectively applied to a previous exam as a “reality check” between the modified-Angoff approach and the historic pass rate. Because all exams have some measurement error, the provisional cut score or standard can be adjusted using the standard error of measurement (SEM). A candidate's theoretic true score falls somewhere within an error band of their estimated score. Using the SEM to adjust the provisional pass point up provides confidence that less than competent candidates who should fail will not pass due to measurement error. Using the SEM to adjust the provisional pass point down provides confidence that competent candidates who should pass will not fail because of measurement error. The operational pass point is set after a final decision about whether to adjust the provisional pass point. The pass point then becomes the criterion-referenced standard or percentage of items that a candidate would need to get correct in order to pass. Each year the percent of items that need to be answered correctly is adjusted for the difficulty of the exam as compared with the standard setting exam. Setting the criterion standard for each exam is performed every 3 to 5 years.
We appreciate that Dr Adkins drew attention to the ABPath Honor Code and the use of “remembrances” for exam preparation. Candidates must agree not to cheat (ie, not to retain, copy, disclose, discuss, share, reveal, distribute, or use for exam preparation examination materials, including memorized, reconstructed, and recalled items). Violation of the Honor Code is a serious breach of professionalism. When a violation occurs, the board can terminate the exam, invalidate the results, withhold or revoke certification, ban from future examinations, or other appropriate action. As the author states, it is unknown how much “remembrances” are used for board preparation, but we expect that our candidates have the integrity to respect the certification process and honor our profession.
The decrease in repeat test takers is also probably in part due to the change in the period of board eligibility. Before 2008, candidates had 3 years of qualification from the year they applied and could get another 3 years eligibility if they had been in active practice for 2 of the prior 3 years. In 2008, the period of board eligibility was changed to 5 years from the year of completion of training.
One of the ABPath values is transparency. We are open and responsive in our communications and collaborations with stakeholders. The article by Dr Adkins supports this transparency in that he has no relationship with the ABPath other than to be a proud diplomate.
Dr Johnson is the chief executive officer of the American Board of Pathology.