ABSTRACT:
Medical licensing examinations are cornerstones that uphold a standardized level of competence among practicing physicians. There are voices that contend the validity of such examinations as accurate measures of physician competency, viewing them as barriers to practice. Research into the predictive efficacy of licensing exams in Canada is still nascent. We attempt to remedy this.
We conducted a historical cohort study of potential factors, including licensing examinations, which might correlate with complaints against family physicians in Alberta using Medical Council of Canada (MCC) and College of Physicians and Surgeons of Alberta (CPSA) data. Logistic regression was used to identify factors associated with non-dismissed complaints (NDCs).
The analyses indicated there are eight NDC predictors, among them the MCC Qualifying Examination (MCCQE) Part I. The regression model was statistically significant, X2 (8, N-539) = 54.23, P<0.0001. In our study, a decrease of one point on the total score on the first attempt is associated with a 0.6% increase in the odds of the physician having an NDC.
The higher the score received on the first MCCQE Part I attempt, the lesser the probability of NDCs in family physicians’ future practice. Our research provides compelling evidence that licensing examinations effectively gauge and predict physician performance, serving as a vital public safeguard.
Background
Medical regulatory authorities (MRAs) hold a critical position in protecting patient care, ensuring physician competence, and assurance of ongoing fitness to practice medicine. However, physicians’ performance in clinical settings is influenced by various factors, including formal medical education, age, patient caseload, and more.1-3 Recognizing the significance of these factors, MRAs must identify and address them as they work to establish and uphold rigorous standards, guidelines, and quality control measures to preserve public safety.
Central to the regulatory framework lies medical licensing examinations: standardized instruments aiming to assess physician competency and guide MRA licensing decisions. Nevertheless, these examinations are subject to ongoing scrutiny with concerns raised regarding their validity and potential to pose barriers to entry into the medical profession.4,5 Evaluating the relationships between licensing examination performance and clinical performance in practice is essential for substantiating the validity and utility of examination scores. The body of research in this area remains limited, particularly with respect to examinations administered in Canada.
A 2020 study conducted by the College of Physicians & Surgeons of Alberta (CPSA) and the Medical Council of Canada (MCC) found that physicians who passed the MCC Qualifying Examination (MCCQE) Part I on their first attempt had significantly fewer complaints against them.6
Others have identified correlations between MCCQE scores and future practice quality, as indicated by peer assessment.7 In the United States, research on specific physician cohorts demonstrated the predictive ability of scores on the United States Medical Licensing Examination (USMLE) for clerkship grades.8-11 However, the correlation between licensing exam scores and patient outcomes remains modest, with limited research considering the entirety of the scoring scale as opposed to dichotomization of the score into pass and failure categories which overlooks nuances in score differences.12,13
Objective
The current research aims to gather and evaluate additional evidence on the relationships between the MCC's examination scores and measures of physician performance in practice. Specifically, we explore whether physicians’ scores on the MCCQE taken between 2014 and 2018, along with personal and practice characteristics, predict subsequent complaints while practising family medicine in Alberta. This study builds on the previous work by De Champlain, et al,6 addressing whether variations in exam scores correlate with variations in future performance. A key improvement over the prior study is the refinement of the data considered; the previous research included all patient complaints, even those potentially dismissed after investigation, which could have skewed the findings since CPSA reports that 25-50% of grievances are dismissed. In contrast, the current study includes only non-dismissed complaints, specifically excluding open, dismissed, or withdrawn complaints.
Methods
Study design and setting
Using MCC and CPSA data, we conducted a historical cohort study of potential factors that might predict complaints against family physicians in Alberta.
Data
The datasets were linked by a third party, the Canadian Post-MD Education Registry (CAPER). A data-sharing agreement (DSA) between the CPSA, MCC, and CAPER was signed by all parties. CAPER matched the datasets, deidentified the research data file, and sent it to the MCC for analysis using a secure data transfer protocol.
The MCC dataset included physician scores on their first attempt of the MCCQE Parts I and II within the study period. The MCCQE Part I measures medical knowledge and clinical decision-making abilities at the level expected of a physician entering supervised practice.14 It should be noted the MCCQE Part II ceased to be offered as of 2021.15. However, all physicians included in our study had to pass the MCCQE Part II, which assessed clinical skills at a level expected of physicians entering independent practice in Canada.
The CPSA dataset had two main components: sociodemographic and practice (registration) variables, and a complaints outcome variable. The CPSA registration data is sourced from two main CPSA databases, the annual Renewal Information Form (RIF)16 database and the Database of the College (DOC). In Alberta, registered physicians are mandated to complete their annual RIF, providing or updating demographic, contact, practice, and educational details (Health Professions Act, Section 34(4), Province of Alberta, as of April 1, 2023). The DOC integrates live data, combining physicians’ information from the RIF, along with complaint data from the CPSA's Professional Conduct Department.
Study sample
Physicians who attempted the MCCQE Part I between 2014 and 2017 and the MCCQE Part II between 2015 and 2018 were included in the study sample. These specific timeframes were chosen due to the comprehensive process of score equating applied to the exam scores during these periods, ensuring comparability over time. Score equating is a meticulous statistical procedure utilized to adjust scores from different versions of an exam, accounting for variations in difficulty and content, thereby enabling meaningful comparisons across different administrations.17 It is best practice to avoid comparisons of exam scores over time when the exam content or procedures change substantially, the reason for which we limit the study to these timeframes. Records with missing values in the outcome variables and covariate variables in the regression model were excluded from the study.
Outcomes
The final model focused on a single outcome: the presence of non-dismissed complaints (NDCs). CPSA maintains a comprehensive database of complaint information for all physicians registered in Alberta. According to the CPSA complaint process as conducted by the Professional Conduct Department, NDCs exclude open, dismissed, or withdrawn complaints.18 Open complaints include those that have yet to be investigated, as well as those currently under investigation before the regulatory body makes a formal decision on the case. For instance, NDCs encompass outcomes such as acknowledgements of receipt of complaint, advice to physicians, educational activities, and assessments. This paper presents the predictive analyses for NDCs. The complaints outcome variable was coded as 1 if physicians had at least one NDC vs. 0 for physicians with no NDCs.
Data analyses
Logistic regression was performed to ascertain the effects of several independent variables on the likelihood that a physician would have an NDC filed against them. A p-value of 0.1 was selected as a less stringent threshold of statistical significance suitable for the relatively small sample size while still allowing for detecting a trend in the relationship between the dependent and independent variables. To mitigate collinearity (a situation in statistical models where two or more independent variables are highly correlated), a correlation matrix was studied to ensure the reliability and interpretability of the model.
Ethics approval
Ethical approval was received from the Health Research Ethics Board at the University of Alberta (Pro00098794).
Results
The predictors of an NDC included years since CPSA registration, an indication of practice location as rural or non-rural, having hospital privileges through Alberta Health Services (AHS), whether a physician was accepting new patients, the number of patients seen on a typical workday, physician gender, international medical graduate (IMG) vs Canadian medical graduate (CMG) designation, and the MCCQE Part I score.
The original dataset included 1201 physicians; however, only 667 had MCCQE Part I and Part II results. Following data cleaning for complete cases analysis, the number was reduced to 539 physicians with full data records; 38 of them had at least one NDC. Analyses were conducted using SAS software version 9.4.
Table 3 presents the coefficients of logistic regression.
The logistic regression coefficients give the change in the log odds of the outcome for a one-unit increase in the predictor variables. Physicians who obtained lower scores on their first attempt at the MCCQE Part I were more likely to have NDCs filed against them. For every one-point change in the score, the physician's odds of complaints increased by 0.006 (1-0.994). This corresponds to 0.6% change in odds.
Discussion
We found that some demographics (physician gender, IMG vs. CMG), as well as practice variables (years of registration with CPSA, number of patients seen on a typical workday, practice location, AHS/hospital privileges, and accepting new patients) were predictors for physicians who may have complaints filed against them. We also learned the score obtained on the MCCQE Part I predicts the likelihood of complaints for family physicians in Alberta later in practice.
Overall, a higher (better) score on the MCCQE Part I translates to a lesser likelihood of future complaints in practice. In our study, a decrease of 100 points on the MCCQE Part I total score on the first attempt, which corresponds to one standard deviation on the score scale, is associated with a 60% increase in the odds of having at least one non-dismissed complaint. These results emphasize the relationship between physician competence (scores on an exam) and physician performance in practice1 and align with findings of other studies on licensing examinations.4,5,11,19,20
Like the 2020 study which looked at the pass/fail decision of the MCCQE Parts I and II, we did not find that MCCQE Part II scores were predictors of an NDC. However, only candidates who had passed the MCCQE Part I were allowed to attempt the MCCQE Part II. Only those who passed the MCCQE Part II would be allowed to practise medicine in Canada. As stated by Wenghofer, “establishing relationships between examination scores (or decisions) and future performance as a physician, which can be attenuated because lower ability candidates may never receive licenses to practise, is difficult.”21
Our findings also revealed that being a female physician and having AHS/hospital privileges decreased the likelihood of having an NDC. More specifically, our results indicated that female physicians are 45% less likely to have an NDC compared to their male counterparts. While the male sex has been shown to be a risk factor for complaints in the literature,22-24 it remains unclear as to why having hospital privileges decreases the chances of having an NDC. We postulate that having hospital privileges might mean a physician must undergo further quality assurance and quality improvement processes mandated by the health authority, thus potentially mitigating the likelihood of NDC occurrences.
Conversely, accepting new patients, practicing in rural locations, and being an IMG were factors associated with an increased likelihood of NDCs. This aligns with existing research. A scoping review of the risks to health professionals’ performance indicated that being an internationally trained physician was a risk factor for performance in practice.25 Similarly, a qualitative study of the risks and protective factors for physician performance identified isolated and rural practice locations as potentially risky for physician performance.2
The notion that accepting new patients into their practice is a risk factor for NDCs is puzzling. It is plausible that it increases the probability of encountering compatibility issues between the physician and their patients. Further investigations are warranted to shed light on this dynamic.
The positive coefficients observed for both years since registration with the CPSA and the number of patients seen in a typical workday suggested that each additional year or patient case is correlated with an increased likelihood of facing NDCs. This may be an intuitive finding since those who see more patients and have been in practice longer may have had more interactions that could have led to more complaints.
Future directions
The authors acknowledge the introduction of a new blueprint in 2018 that significantly changed the MCCQE Part I. Given these changes, exploring whether the updated MCCQE Part I will predict NDCs, and other outcome measures would be advisable. Conducting such a longitudinal study would be a multi-year project as it would require accumulating data over several years, including collecting sufficient practice data.
Limitations of the study
This study included only family physicians from one province (Alberta). Without a national-level database on physician registration information, it is much more challenging to collect and consolidate data from various jurisdictions. It is hoped that national initiatives, such as the National Registration of Physicians (NRP), will create more opportunities for pan-Canadian analyses.26
We acknowledge that while complaints serve as a useful proxy for physician performance, they have their limitations. Complaints cover a wide range of issues, including practice management, medical reporting, quality of care, third-party issues, systemic concerns, and ethics, but they may not encompass all aspects of a competent physician. To provide a more comprehensive assessment of physician performance, it would be beneficial to include other outcome measures, such as length of stay, mortality rates, and patient satisfaction.13,20,21,27
Conclusion
While many countries, including Canada, face an unprecedented shortage in the healthcare workforce, some educators and decision-makers advocate for removing national licensing examinations, viewing them as barriers to practice. This contrasts with other jurisdictions’ efforts to establish robust national licensing exams.28,29 However, while examinations should not be the sole determinant of licensing decisions, they remain an essential component of a comprehensive regulatory framework aimed at safeguarding patient care and maintaining the integrity of the medical profession.
Ample research studies, including our findings, underscore the importance of licensing examinations as indicators of a physician's future performance in practice.21 From an assessment perspective, such evidence supports the validity argument that these examinations measure physicians’ knowledge and competencies related to safe and effective patient care. This is significant, even if the evidence is often modest. These exams serve multiple purposes in such a regulatory framework, including the standardized assessment of medical trainees, monitoring of teaching in medical schools (often necessary for accreditation of medical schools), and as a tool to predict future physician performance in practice.
Nonetheless, it is crucial to continue research and dialogue to refine and enhance the use of examinations within the broader context of physician licensure and patient safety. This ongoing effort will help ensure that licensing examinations contribute meaningfully to the health-care system while addressing any limitations and challenges.
References
Funding/support: This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
Other disclosures: N/A
Ethics statement: This study was approved by the University of Alberta's Health Research Ethics Board Study ID: Pro00098794.
Author contributions: Study concept and design (IB, NK, NA, MM); Acquisition of data (NA, NH, KK); Analysis and/or interpretation of data (RN, IB, NA, NH, MM); Drafting of the manuscript (IB, NK, IH, HH, KK); Critical revision of the manuscript for important intellectual content (IB, NK, MM)