ABSTRACT
Medical licensing authorities and other entities utilize and rely on the United States Medical Licensing Examination (USMLE) as a standardized, valid and reliable tool to assess physicians’ knowledge and skills. As such, engaging in irregular behavior during the USMLE process can have a broad and damaging impact on an individual’s ability to complete the USMLE sequence and subsequently obtain a medical license in the United States. While there are also repercussions for the USMLE program and entities overseeing medical students and physicians, the risk to the public of being cared for by someone who did not pass a medical licensing examination by his or her own merit is of great concern. This study reviews data about individuals who engaged in irregular behavior, common sanctions taken against them and their ability to ultimately practice medicine in the United States.
Using data from the Federation of State Medical Boards (FSMB) and the National Board of Medical Examiners (NBME), individuals with findings of irregular behavior as part of the USMLE application and/or testing process between 2006 to 2015 were identified, along with their corresponding demographic, examination, sanction and licensure outcome data.
A total of 165 individuals were found to have engaged in 170 incidents of irregular behavior. The majority of individuals were male (63%, n = 101), international medical graduates (69%, n = 112) and had a mean age of 33 years old (SD = 8 years, n = 161) at the first incident or finding of irregular behavior. The two most common types of irregular behavior were falsified information (31%, n = 53) and security violations (27%, n = 46). Most incidents (86%, n = 146) were reported to the FSMB Physician Data Center (PDC) and 68% (n = 116) involved a bar from taking the USMLE for a period of time. Only 26% (n = 43) of individuals ultimately passed the entire USMLE sequence and 16% (n = 27) obtained a full, unrestricted medical license in the U.S. by 2019.
To help maintain the integrity of a key component for initial licensure in the United States, there is a continued need for rigorous enforcement and safeguarding of USMLE examination applications, content, testing conditions and score reports. Individuals who plan on taking the USMLE should become familiar with USMLE rules and penalties regarding irregular behavior, including the serious implications of such behavior that can severely diminish their ability to practice medicine in the United States. Entities using USMLE information for licensure or admission into medical-related programs need to diligently ensure authentication of USMLE documents and carefully consider if individuals who have engaged in irregular behavior are qualified to practice medicine.
Introduction
From the late 19th century into the mid-20th century, medical licensing authorities in the United States had a direct role in the assessment of physicians through the development and administration of state-based examinations for licensure. Throughout the latter half of the 20th century, this role transitioned and was outsourced to national test development organizations in order to ensure a standardized, valid and reliable assessment of physicians. Today, all state medical and osteopathic boards require the United States Medical Licensing Examination (USMLE) or the Comprehensive Osteopathic Licensing Examination-USA (COMLEX-USA) as the assessment of physician knowledge and skills for the purposes of licensure. These examinations not only meet the state boards’ statutory responsibility to assess physicians for purposes of licensure but fulfill the boards’ primary mission to protect the public by ensuring that only qualified practitioners are given a license to practice. Individuals who engage in irregular behavior related to the USMLE, therefore, represent a serious threat to the integrity of the exam, their ability to practice medicine and public safety.
The USMLE comprises a three-step examination sequence (Step 1, Step 2 Clinical Knowledge, Step 2 Clinical Skills and Step 3) for initial medical licensure in the United States open to students and graduates of MD- and DO-degree granting medical schools. The USMLE program has conducted more than 3 million test administrations since its inception in 1992. Considering the critical role that the USMLE serves, the organization has the responsibility to ensure the secure development, administration and scoring of the examination.1 One of the many ways the USMLE program does this is through its processes for reviewing cases of alleged irregular behavior and taking appropriate actions. Although cases involving alleged irregular behavior related to the USMLE are rare, the negative ramifications extend beyond the individual and the USMLE program into the medical community and ultimately the American public.
The USMLE program defines irregular behavior as any action by applicants, examinees or others that could compromise the validity, integrity or security of the USMLE, such as seeking or obtaining unauthorized access to examination materials; providing false information on an examination application; impersonating an examinee; possessing unauthorized items during the examination; altering or misrepresenting examination scores; unauthorized reproduction of examination content; and posting or discussing examination content on any website.2
The USMLE program is informed of suspected irregular behavior by a variety of paths — e.g., proctor reports from testing centers; staff at the National Board of Medical Examiners (NBME), the Federation of State Medical Boards (FSMB) and the Educational Commission for Foreign Medical Graduates (ECFMG) as part of regular, ongoing registration activities; staff monitoring of online websites, chatrooms and other sources; tips to the program; and inquiries from medical education administrators. Reports that warrant further review are referred to the USMLE Committee for Individualized Review (CIR), which is composed of physician and public members appointed by the NBME, FSMB and ECFMG. Once a case is referred to the CIR, the individual is notified of the basis for the allegation and is given an opportunity to submit relevant information to be considered by the CIR. Individuals may secure legal counsel and/or make personal appearances before the CIR to give testimony under oath.
If the CIR finds that irregular behavior has occurred, official USMLE score reports for that examination are annotated with a finding of irregular behavior. Additionally, all official USMLE transcripts sent out for the individual include an annotation (stating IRREGULAR BEHAVIOR in bold capital letters), as well as a short descriptor of the irregular behavior (e.g., falsified score, security violation). The CIR determination letter (i.e., the letter to the individual regarding the basis for, outcome of and actions taken from the CIR’s review) is also attached anytime this transcript is shared.
In addition to annotating the USMLE transcript, the CIR can impose a range of additional actions, such as barring individuals from the USMLE for a specified period of time, reporting findings and sanctions to the FSMB’s Physician Data Center (PDC) and directly informing relevant third parties of its findings. If the finding of irregular behavior is reported to the PDC, this information is also proactively provided as part of the official USMLE transcript through an FSMB PDC Practitioner Profile. Independent of the transcript process, state medical boards and other entities (e.g., residency programs, hospitals, employers) are also notified of irregular behavior findings reported to the PDC any time they query the PDC for that individual. When deciding on additional sanctions, the CIR considers the precedent, if any, established in other similar cases.
Unfortunately, unethical testing behavior is not new to the medical licensing examination. Various forms of cheating have occurred since the advent of these exams in the late 19th century.3 Such behaviors then and now fall within a broader context of unprofessional behavior. For example, in a review of 51 articles published between 1980 to 2014, between 5% to 15% of medical students and residents engaged in plagiarizing, cheating on exams and/or listing fraudulent publications on residency and fellowship applications.4 Specifically related to the USMLE, a previous study identified that 433 individuals engaged in irregular behavior on those exams between 1992 to 2006.5 Looking at performance on the COMLEX-USA Level 2-PE, a licensing exam available for osteopathic physicians, approximately 0.2% of the examinees fail due to fabricating or misrepresenting their clinical findings on their post-encounter notes.6,7,8 An up-to-date review of individuals who engaged in irregular behavior related to the USMLE will help to understand the characteristics related to this behavior and serve as an effort to dissuade others who are considering engaging in this unprofessional activity.
This study is designed to document outcomes for individuals with findings of irregular behavior between 2006 to 2015, specifically to: 1) summarize the demographic characteristics of individuals who engaged in irregular behavior, 2) categorize the most common bases for irregular behavior, 3) identify sanctions imposed, 4) determine the percentage of individuals with irregular behavior who completed the USMLE sequence and obtained a full, unrestricted medical license in the United States and 5) summarize the disparate licensure outcomes using a case study of an individual who engaged in irregular behavior.
Methods
Using data from the FSMB and the NBME, 165 individuals with findings of irregular behavior as part of the USMLE Steps 1, 2 Clinical Knowledge (CK), 2 Clinical Skills (CS) and/or 3 application and/or testing process between 2006 to 2015 were identified. The end date of 2015 was selected for the study to allow time for individuals to have obtained a medical license in the United States by the end of 2019.
Individuals with administrative irregular behavior were identified based on the date of the last administration day of the respective examination. Administrative irregular behavior is behavior that occurs in connection with an examination that was administered, such as bringing or taking prohibited material into or out of the testing center. Individuals with non-administrative irregular behavior were identified based on the date of the CIR determination letter since there was no examination date to reference. Non-administrative irregular behavior is behavior that occurs outside of an examination administration or in connection with an examination that was either never administered or not completed, such as falsifying information on a USMLE application.
Variables reviewed included individual characteristics (i.e., gender, location of medical school, age at time of irregular behavior), the type of irregular behavior that occurred, if the irregular behavior incident was reported to the PDC, length of bar from taking the USMLE, if the individuals completed the full USMLE sequence (i.e., passed Step 3) and if they obtained a medical license in the United States by 2019.
Results
A total of 165 individuals were found to have engaged in 170 incidents of irregular behavior between 2006 and 2015 (75 findings of administrative irregular behavior and 95 findings of non-administrative irregular behavior). It is possible for individuals to have more than one finding of irregular behavior (e.g., second finding of irregular behavior for a subsequent examination or to have a finding of both administrative irregular behavior and non-administrative irregular behavior); thus, the difference in total number of individuals versus total number of irregular behavior findings. Although it is hard to calculate the occurrence of irregular behavior because these instances can occur outside of the examination, as a reference point more than 138,000 and 140,000 USMLE exams were administered each year in 2006 and 2015, respectively.*,9
Demographics
The majority of individuals were male (63%, n = 101) and international medical graduates (IMGs) (69%, n = 112) (Table 1). A cross-tabulation of gender and location of medical school shows that 44% of individuals with irregular behavior were male IMGs (n = 71), followed by female IMGs (24%, n = 39), male U.S./Canadian graduates (18%, n = 29) and female U.S./Canadian graduates (13%, n = 21) (Table 2). The mean age at the first incident or finding of irregular behavior from 2006 to 2015 was 33 years old (SD = 8 years, n = 161) (Table 1). By comparison, descriptive statistics from a study of demographic differences in USMLE scores from 2010 to 2015 for U.S. and Canadian medical graduates showed that approximately 48% of individuals were female and had an average age of 25 years old at the first Step 1 attempt.10
Bases for Irregular Behavior
The most common irregular behavior was falsified information (31%, n = 53), followed by security violations (27%, n = 46), procedural violations (16%, n = 27), falsified scores (11%, n = 19), ineligibility (11%, n = 18) and disruptive behavior or unauthorized assistance (4%, n = 7) (Table 3). Example behaviors that fit within each category are provided in Table 3.
Irregular Behavior by Step Examination
Since non-administrative irregular behavior is not associated with an examination that has been administered or scored, the analysis of irregular behavior by Step examination only focused on administrative irregular behavior. A review of administrative irregular behavior indicates the highest percentages of incidents were in connection with Step 1 (33%, n = 25) and Step 2 CS (n = 32%, n = 24), followed by Step 3 (20%, n = 15) and Step 2 CK (15%, n = 11) (Table 4).
Reporting of Irregular Behavior to the FSMB PDC
A review of the 170 irregular behavior incidents between 2006 to 2015 found that the majority (86%, n = 146) were reported to the FSMB PDC (Table 5).**
Barring Individuals from Future USMLE Examinations
Over two-thirds of irregular behavior incidents resulted in a time-specific bar from taking future USMLE examinations (68%, n = 116). Bars ranged from six months to five years. The most common bar length was for three to five years (47%, n = 54), with the clear majority of these bars being at three years.
In addition to imposing a bar from future USMLE examinations, the CIR included the additional stipulation that future access to the USMLE would only be granted upon request from a state medical board that was fully informed of the circumstances regarding the irregular behavior for 35 of the 116 incidents that received bars (30%) (Table 5). Table 6 offers reasons for this added stipulation. Throughout the history of the USMLE program, no state medical board has yet requested that an individual with this stipulation be allowed to take the USMLE. As a result, none of these individuals have been able to complete the USMLE sequence, nor have they been able to obtain licensure in the United States.
Completion of USMLE Sequence
As noted in Table 1, of the 165 individuals found to have engaged in irregular behavior from 2006 to 2015, 26% (n = 43) have gone on to pass the full USMLE sequence.
Licensure Outcomes
Of the 165 individuals identified with irregular behavior, only 16% (n = 27) had obtained a full, unrestricted medical license in the United States by 2019 (limited, special, temporary and training licenses were excluded where identifiable) (Table 1). A review of those licensed revealed that 52% had a security violation (n = 14), 30% (n = 8) had a procedural violation and 19% had either disruptive behavior or falsified information (n = 5).
As outlined in the case study below, individuals with irregular behavior may have varying degrees of difficulty in obtaining a medical license to practice in the United States.
Discussion
Engaging in USMLE irregular behavior is detrimental not only to the individual who may be delayed or permanently restricted from practicing medicine, but also to the USMLE program, entities that oversee the regulation and employment of physicians and ultimately the public, which could be exposed to practicing physicians who have not met medical licensing examination requirements on their own merit.
A finding of irregular behavior introduces a substantial hurdle to a physician’s prospective career. Although some individuals in this study (n = 27) were able to successfully complete the full USMLE sequence and ultimately obtain a medical license, for the majority of this cohort, the outcome is less clear. It is possible that some individuals, particularly with more recent irregular behavior findings, have not yet had sufficient time to complete the USMLE sequence and gain access to a residency training program. It also seems likely that some individuals who are unable to complete the full USMLE sequence opt for a different career path or try to pursue medical licensure and practice in another country.
Loss of examination material as a result of unauthorized sharing of content comes at a significant cost to the USMLE program, not only financially but also in terms of time, human and physical resources. Exposed items must be removed from the examinations and the item pool, and replacement items must be written, reviewed and pre-tested before they are implemented as scored items. The USMLE program relies on more than 300 volunteers from the academic, medical, licensing and practicing physician communities across the United States to write test items, with individuals investing many hours to write a single, valid and reliable item. Therefore, the cost is real and tangible when content is removed from an examination due to misconduct.
An examinee, J. Doe,* was found with study materials in the testing room during the administration of a computer-based USMLE Step examination. Subsequently, the USMLE CIR determined that this individual had engaged in irregular behavior and reported its finding to the FSMB PDC. Dr. Doe ultimately completed the USMLE sequence and applied for a medical license in three states, with disparate outcomes.
Dr. Doe applied for and was granted a full, unrestricted license in State A. This is the same state in which Dr. Doe had already obtained a training license and completed residency training.
Several years later, Dr. Doe applied for a license in State B. Ultimately, Dr. Doe withdrew the license application from this state. Although there is no publicly available information about why the application was withdrawn, it is possible that it was related to engaging in irregular behavior during the USMLE examination.
Several years after withdrawing the license application in State B, Dr. Doe applied for a license in State C. Ultimately, Dr. Doe’s license application in State C was permanently denied. According to publicly available information, the basis for the denial was the irregular behavior Dr. Doe engaged in during the USMLE examination. Initially issued a citation by the board, Dr. Doe was later issued an order of permanent denial due to having committed fraud during the administration of an examination for licensure or deception in applying for a license and for behavior that constitutes a violation of the American Medical Association Principles of Medical Ethics.
Although the case study of this physician highlights the variety of licensing outcomes individuals found to have engaged in irregular behavior can experience, it is likely that they will face challenges obtaining and maintaining an unrestricted license to practice medicine. This case demonstrates the negative impact that engaging in irregular behavior in connection with a medical licensing examination can have on a physician’s career — even years after the event.
* A fictious name was used to protect the identity of the individual.
Security violations (27%, n = 46) were the second most common reason for irregular behavior in this study after falsified information (31%, n = 53) and likely account for individuals who solicited and/or posted examination content. The USMLE program regularly monitors the internet for propriety or copyrighted content and has seen an increase in these types of behaviors over the years, likely due to the increased use of the internet. A previous irregular behavior study showed only 21 cases related to content posted on the internet from 1992 to 2006. Rather, the most common irregular behavior then was timing violations (37%, n = 163) involving examinees marking on the answer sheet after the proctor called time on USMLE paper and pencil examinations, a format that is no longer offered.5
While all irregular behavior is a clear breach in conduct, the nature of the behavior largely dictates the severity of consequences. For example, this study found that 68% of the incidents involved a bar from taking future USMLE examinations for a specified period of time and 86% were reported to the FSMB PDC, whereas from 1992 to 2006, only 28% of the incidents involved a bar and 33% were reported to the PDC.5 This difference is likely explained, in part, by the nature of the behaviors seen; in particular, the recent increase in sharing of examination content online is deemed as more egregious by the CIR while other behaviors, such as timing violations, do not typically merit bars or additional sanctions.
The high-stakes nature of the USMLE is likely a factor in irregular behavior for any USMLE Step, but individuals may have different motivations for engaging in irregular behavior depending on which examination they are taking. For example, given the historical utilization of and reliance on Step 1 scores for residency selection in the United States, examinees likely feel additional pressure to not only pass Step 1, but also achieve the highest score possible. A review of the CIR determination letters — which provide a summary of the examinee’s response to the irregular behavior allegations — indicates that aspirations of getting into a desired specialty area or program has been cited by examinees as a reason for falsifying examination scores. The higher presence of administrative irregular behavior associated with Step 1 could also be attributed to the fact that this is typically the first examination that individuals take; therefore, they may not be as knowledgeable or mindful of the rules of the examination or the implications for breaching these rules. In a traditional U.S. four-year medical education program, Step 1 is typically taken at the end of the second year, a point in time in students’ professional development when most have limited knowledge about state licensing authorities and licensure. As such, the repercussions of engaging in irregular behavior during an examination for licensure is likely not at the forefront of Step 1 examinees’ preparations.
A review of the CIR determination letters also indicates that Step 3 scores have been falsified based on a need to present a passing score in order to advance from year 2 to year 3 in graduate medical education training or to be eligible for licensure in the United States. Examinees have also cited a desire to avoid displeasing family as the rationale for falsification of examination scores — whether from a fail to a pass or from a pass to a higher pass.
Specific examinee sub-populations may also be vulnerable to other pressures. For example, IMGs are disproportionately represented in this irregular behavior cohort compared to the general USMLE testing population. While each situation is unique, the need to pass Steps 1 and 2 to be eligible to obtain a J-1 Visa and also pass Step 3 for a H-1B Visa for entry into graduate medical education training in the United States,11, 12 as well as the potential difficulty of being matched into a U.S. graduate medical education program,13 may put increased pressure on IMGs. These scenarios may incentivize individuals, for example, to falsify examination scores to report a higher score or fewer attempts needed to pass in order to be more competitive residency candidates or to obtain the necessary visas to complete their education and training in the United States. Cultural differences may also be a contributing factor. IMGs may not have the same support systems from their medical schools as U.S. students do, thus making them more reliant on networking, chat rooms, study groups and other measures to prepare for the USMLE. Examination preparation in general may also be viewed as more of a communal or group endeavor, in which sharing of information is encouraged or at least viewed favorably, but such behavior can put individuals at risk of improperly sharing examination content. Admittedly, the motives offered in this paper as to why individuals engage in irregular behavior are not exhaustive, and the reasons given by the accused may still lack transparency. Furthermore, many individuals alleged to have engaged in irregular behavior do not admit to it, even when confronted with irrefutable, direct evidence of their guilt.
There are several limitations to our study. First, while the CIR determination letters offer valuable insight as to why individuals engaged in the irregular behavior, the study is only able to offer reasons to the extent that the individuals chose to respond and explain the allegations. Second, while the case study helps illustrate the variable success physicians will likely face in obtaining a license to practice medicine after a finding of irregular behavior, the study is limited in its ability to explain why each state board acted in the manner that it did due to lack of publicly available information. Next, explanations by, and outcomes involving, individuals who engaged in irregular behavior should be interpreted in light of a relatively small population. This, as well as the sensitive nature of the allegations, prohibited full disclosure of all details and outcomes of the irregular behavior.
To help maintain the integrity of a key component for initial licensure in the United States, there is a continued need for stringent enforcement and safeguarding of the USMLE examination applications, content, testing conditions and score reports. Due to the high-stakes nature of this exam, some examinees may be tempted to cheat, even for people who generally have high standards of integrity. This study can provide further visibility of the security standards the USMLE has in place and the actions taken when someone breaches these standards, which may help dissuade any individual who is contemplating engaging in irregular behavior. Individuals preparing to take the USMLE should become familiar with its rules and penalties regarding irregular behavior in order to avoid the immediate and long-term consequences that likely will diminish their prospects of practicing medicine in the U.S. Since most irregular behavior occurs in relation to the Step examinations taken during undergraduate medical education, medical schools are encouraged to help make their students aware of the USMLE as a medical licensing examination and the stakes involved when examinees engage in irregular behavior.
A finding of irregular behavior allows the USMLE program to inform licensing authorities, medical schools, graduate medical education programs and other interested entities about individuals who have engaged in behavior that could have compromised the integrity of a medical licensing examination. Therefore, this study highlights the need for these entities to carefully review the nature and circumstances of the irregular behavior and the sanctions imposed by the CIR and to reach out to the USMLE program for further explanation if needed. The prevalence of irregular behavior related to falsification of information also highlights the need for organizations and programs overseeing medical students and physicians to require and accept only official USMLE transcripts for purposes of verifying USMLE performance. Evidence of irregular behavior may prompt consideration of whether these individuals are a good fit for their program and/or are qualified to practice medicine.
Acknowledgements
A sincere thank you to Sharon Ray for helping gather data for this project.
References
* Step 1 and Step 3 administrations calculated close to the calendar year. Step 2 CK and CS administrations calculated from July 2006 – June 2007 and July 2014–June 2015, respectively.
** Reporting to PDC is not automatically assessed upon a finding of irregular behavior, but rather based on the severity of the incident(s) and precedent from similar cases.
About the Authors
Frances E. Cain, MPA, is Director, Assessment Services, at the FSMB.
Katie Arnhart, PhD, is Senior Research Analyst, Research and Data Integration, at the FSMB.
Aaron Young, PhD, is Vice President, Research and Data Integration, at the FSMB.
David Johnson, MA, is Chief Assessment Officer at the FSMB.