Background:

Medical licensing examinations are cornerstones that uphold a standardized level of competence among practicing physicians. There are voices that contend the validity of such examinations as accurate measures of physician competency, viewing them as barriers to practice. Research into the predictive efficacy of licensing exams in Canada is still nascent. We attempt to remedy this.

Methods:

We conducted a historical cohort study of potential factors, including licensing examinations, which might correlate with complaints against family physicians in Alberta using Medical Council of Canada (MCC) and College of Physicians and Surgeons of Alberta (CPSA) data. Logistic regression was used to identify factors associated with non-dismissed complaints (NDCs).

Results:

The analyses indicated there are eight NDC predictors, among them the MCC Qualifying Examination (MCCQE) Part I. The regression model was statistically significant, X2 (8, N-539) = 54.23, P<0.0001. In our study, a decrease of one point on the total score on the first attempt is associated with a 0.6% increase in the odds of the physician having an NDC.

Conclusion:

The higher the score received on the first MCCQE Part I attempt, the lesser the probability of NDCs in family physicians’ future practice. Our research provides compelling evidence that licensing examinations effectively gauge and predict physician performance, serving as a vital public safeguard.

Medical regulatory authorities (MRAs) hold a critical position in protecting patient care, ensuring physician competence, and assurance of ongoing fitness to practice medicine. However, physicians’ performance in clinical settings is influenced by various factors, including formal medical education, age, patient caseload, and more.1-3  Recognizing the significance of these factors, MRAs must identify and address them as they work to establish and uphold rigorous standards, guidelines, and quality control measures to preserve public safety.

Central to the regulatory framework lies medical licensing examinations: standardized instruments aiming to assess physician competency and guide MRA licensing decisions. Nevertheless, these examinations are subject to ongoing scrutiny with concerns raised regarding their validity and potential to pose barriers to entry into the medical profession.4,5  Evaluating the relationships between licensing examination performance and clinical performance in practice is essential for substantiating the validity and utility of examination scores. The body of research in this area remains limited, particularly with respect to examinations administered in Canada.

A 2020 study conducted by the College of Physicians & Surgeons of Alberta (CPSA) and the Medical Council of Canada (MCC) found that physicians who passed the MCC Qualifying Examination (MCCQE) Part I on their first attempt had significantly fewer complaints against them.6 

Others have identified correlations between MCCQE scores and future practice quality, as indicated by peer assessment.7  In the United States, research on specific physician cohorts demonstrated the predictive ability of scores on the United States Medical Licensing Examination (USMLE) for clerkship grades.8-11  However, the correlation between licensing exam scores and patient outcomes remains modest, with limited research considering the entirety of the scoring scale as opposed to dichotomization of the score into pass and failure categories which overlooks nuances in score differences.12,13 

The current research aims to gather and evaluate additional evidence on the relationships between the MCC's examination scores and measures of physician performance in practice. Specifically, we explore whether physicians’ scores on the MCCQE taken between 2014 and 2018, along with personal and practice characteristics, predict subsequent complaints while practising family medicine in Alberta. This study builds on the previous work by De Champlain, et al,6  addressing whether variations in exam scores correlate with variations in future performance. A key improvement over the prior study is the refinement of the data considered; the previous research included all patient complaints, even those potentially dismissed after investigation, which could have skewed the findings since CPSA reports that 25-50% of grievances are dismissed. In contrast, the current study includes only non-dismissed complaints, specifically excluding open, dismissed, or withdrawn complaints.

Study design and setting

Using MCC and CPSA data, we conducted a historical cohort study of potential factors that might predict complaints against family physicians in Alberta.

Data

The datasets were linked by a third party, the Canadian Post-MD Education Registry (CAPER). A data-sharing agreement (DSA) between the CPSA, MCC, and CAPER was signed by all parties. CAPER matched the datasets, deidentified the research data file, and sent it to the MCC for analysis using a secure data transfer protocol.

The MCC dataset included physician scores on their first attempt of the MCCQE Parts I and II within the study period. The MCCQE Part I measures medical knowledge and clinical decision-making abilities at the level expected of a physician entering supervised practice.14  It should be noted the MCCQE Part II ceased to be offered as of 2021.15. However, all physicians included in our study had to pass the MCCQE Part II, which assessed clinical skills at a level expected of physicians entering independent practice in Canada.

The CPSA dataset had two main components: sociodemographic and practice (registration) variables, and a complaints outcome variable. The CPSA registration data is sourced from two main CPSA databases, the annual Renewal Information Form (RIF)16  database and the Database of the College (DOC). In Alberta, registered physicians are mandated to complete their annual RIF, providing or updating demographic, contact, practice, and educational details (Health Professions Act, Section 34(4), Province of Alberta, as of April 1, 2023). The DOC integrates live data, combining physicians’ information from the RIF, along with complaint data from the CPSA's Professional Conduct Department.

Study sample

Physicians who attempted the MCCQE Part I between 2014 and 2017 and the MCCQE Part II between 2015 and 2018 were included in the study sample. These specific timeframes were chosen due to the comprehensive process of score equating applied to the exam scores during these periods, ensuring comparability over time. Score equating is a meticulous statistical procedure utilized to adjust scores from different versions of an exam, accounting for variations in difficulty and content, thereby enabling meaningful comparisons across different administrations.17  It is best practice to avoid comparisons of exam scores over time when the exam content or procedures change substantially, the reason for which we limit the study to these timeframes. Records with missing values in the outcome variables and covariate variables in the regression model were excluded from the study.

Outcomes

The final model focused on a single outcome: the presence of non-dismissed complaints (NDCs). CPSA maintains a comprehensive database of complaint information for all physicians registered in Alberta. According to the CPSA complaint process as conducted by the Professional Conduct Department, NDCs exclude open, dismissed, or withdrawn complaints.18  Open complaints include those that have yet to be investigated, as well as those currently under investigation before the regulatory body makes a formal decision on the case. For instance, NDCs encompass outcomes such as acknowledgements of receipt of complaint, advice to physicians, educational activities, and assessments. This paper presents the predictive analyses for NDCs. The complaints outcome variable was coded as 1 if physicians had at least one NDC vs. 0 for physicians with no NDCs.

Data analyses

Logistic regression was performed to ascertain the effects of several independent variables on the likelihood that a physician would have an NDC filed against them. A p-value of 0.1 was selected as a less stringent threshold of statistical significance suitable for the relatively small sample size while still allowing for detecting a trend in the relationship between the dependent and independent variables. To mitigate collinearity (a situation in statistical models where two or more independent variables are highly correlated), a correlation matrix was studied to ensure the reliability and interpretability of the model.

Ethics approval

Ethical approval was received from the Health Research Ethics Board at the University of Alberta (Pro00098794).

The predictors of an NDC included years since CPSA registration, an indication of practice location as rural or non-rural, having hospital privileges through Alberta Health Services (AHS), whether a physician was accepting new patients, the number of patients seen on a typical workday, physician gender, international medical graduate (IMG) vs Canadian medical graduate (CMG) designation, and the MCCQE Part I score.

The original dataset included 1201 physicians; however, only 667 had MCCQE Part I and Part II results. Following data cleaning for complete cases analysis, the number was reduced to 539 physicians with full data records; 38 of them had at least one NDC. Analyses were conducted using SAS software version 9.4.

Tables 1 and 2 present the descriptive statistics of the variables in the final model.

Table 1

Descriptive statistics of continuous variables included in the final model

Descriptive statistics of continuous variables included in the final model
Descriptive statistics of continuous variables included in the final model
Table 2

Descriptive statistics of the categorial variables included in the final model

Descriptive statistics of the categorial variables included in the final model
Descriptive statistics of the categorial variables included in the final model

Table 3 presents the coefficients of logistic regression.

Table 3

Descriptive statistics of the categorial variables included in the final model

Descriptive statistics of the categorial variables included in the final model
Descriptive statistics of the categorial variables included in the final model

The logistic regression coefficients give the change in the log odds of the outcome for a one-unit increase in the predictor variables. Physicians who obtained lower scores on their first attempt at the MCCQE Part I were more likely to have NDCs filed against them. For every one-point change in the score, the physician's odds of complaints increased by 0.006 (1-0.994). This corresponds to 0.6% change in odds.

We found that some demographics (physician gender, IMG vs. CMG), as well as practice variables (years of registration with CPSA, number of patients seen on a typical workday, practice location, AHS/hospital privileges, and accepting new patients) were predictors for physicians who may have complaints filed against them. We also learned the score obtained on the MCCQE Part I predicts the likelihood of complaints for family physicians in Alberta later in practice.

Overall, a higher (better) score on the MCCQE Part I translates to a lesser likelihood of future complaints in practice. In our study, a decrease of 100 points on the MCCQE Part I total score on the first attempt, which corresponds to one standard deviation on the score scale, is associated with a 60% increase in the odds of having at least one non-dismissed complaint. These results emphasize the relationship between physician competence (scores on an exam) and physician performance in practice1  and align with findings of other studies on licensing examinations.4,5,11,19,20 

Like the 2020 study which looked at the pass/fail decision of the MCCQE Parts I and II, we did not find that MCCQE Part II scores were predictors of an NDC. However, only candidates who had passed the MCCQE Part I were allowed to attempt the MCCQE Part II. Only those who passed the MCCQE Part II would be allowed to practise medicine in Canada. As stated by Wenghofer, “establishing relationships between examination scores (or decisions) and future performance as a physician, which can be attenuated because lower ability candidates may never receive licenses to practise, is difficult.”21 

Our findings also revealed that being a female physician and having AHS/hospital privileges decreased the likelihood of having an NDC. More specifically, our results indicated that female physicians are 45% less likely to have an NDC compared to their male counterparts. While the male sex has been shown to be a risk factor for complaints in the literature,22-24  it remains unclear as to why having hospital privileges decreases the chances of having an NDC. We postulate that having hospital privileges might mean a physician must undergo further quality assurance and quality improvement processes mandated by the health authority, thus potentially mitigating the likelihood of NDC occurrences.

Conversely, accepting new patients, practicing in rural locations, and being an IMG were factors associated with an increased likelihood of NDCs. This aligns with existing research. A scoping review of the risks to health professionals’ performance indicated that being an internationally trained physician was a risk factor for performance in practice.25  Similarly, a qualitative study of the risks and protective factors for physician performance identified isolated and rural practice locations as potentially risky for physician performance.2 

The notion that accepting new patients into their practice is a risk factor for NDCs is puzzling. It is plausible that it increases the probability of encountering compatibility issues between the physician and their patients. Further investigations are warranted to shed light on this dynamic.

The positive coefficients observed for both years since registration with the CPSA and the number of patients seen in a typical workday suggested that each additional year or patient case is correlated with an increased likelihood of facing NDCs. This may be an intuitive finding since those who see more patients and have been in practice longer may have had more interactions that could have led to more complaints.

Future directions

The authors acknowledge the introduction of a new blueprint in 2018 that significantly changed the MCCQE Part I. Given these changes, exploring whether the updated MCCQE Part I will predict NDCs, and other outcome measures would be advisable. Conducting such a longitudinal study would be a multi-year project as it would require accumulating data over several years, including collecting sufficient practice data.

Limitations of the study

This study included only family physicians from one province (Alberta). Without a national-level database on physician registration information, it is much more challenging to collect and consolidate data from various jurisdictions. It is hoped that national initiatives, such as the National Registration of Physicians (NRP), will create more opportunities for pan-Canadian analyses.26 

We acknowledge that while complaints serve as a useful proxy for physician performance, they have their limitations. Complaints cover a wide range of issues, including practice management, medical reporting, quality of care, third-party issues, systemic concerns, and ethics, but they may not encompass all aspects of a competent physician. To provide a more comprehensive assessment of physician performance, it would be beneficial to include other outcome measures, such as length of stay, mortality rates, and patient satisfaction.13,20,21,27 

While many countries, including Canada, face an unprecedented shortage in the healthcare workforce, some educators and decision-makers advocate for removing national licensing examinations, viewing them as barriers to practice. This contrasts with other jurisdictions’ efforts to establish robust national licensing exams.28,29  However, while examinations should not be the sole determinant of licensing decisions, they remain an essential component of a comprehensive regulatory framework aimed at safeguarding patient care and maintaining the integrity of the medical profession.

Ample research studies, including our findings, underscore the importance of licensing examinations as indicators of a physician's future performance in practice.21  From an assessment perspective, such evidence supports the validity argument that these examinations measure physicians’ knowledge and competencies related to safe and effective patient care. This is significant, even if the evidence is often modest. These exams serve multiple purposes in such a regulatory framework, including the standardized assessment of medical trainees, monitoring of teaching in medical schools (often necessary for accreditation of medical schools), and as a tool to predict future physician performance in practice.

Nonetheless, it is crucial to continue research and dialogue to refine and enhance the use of examinations within the broader context of physician licensure and patient safety. This ongoing effort will help ensure that licensing examinations contribute meaningfully to the health-care system while addressing any limitations and challenges.

1.
Rethans
J-J
,
Norcini
JJ
,
Baron-Maldonado
M
, et al.
The relationship between competence and performance: implications for assessing practice performance
.
Med Educ.
2002
;
36
(
10
):
901
9
.
doi:
10.1046/j.1365-2923.2002.01316.x
2.
Kain
NA
,
Hodwitz
K
,
Yen
W
,
Ashworth
N.
Experiential knowledge of risk and support factors for physician performance in Canada: A qualitative study
.
BMJ Open.
2019
;
9
(
2
):e023511.
doi:
10.1136/bmjopen-2018-023511
3.
Yen
W
,
Thakkar
N.
State of the science on risk and support factors to physician performance: a report from the pan-Canadian physician factors collaboration
.
J Med Regul.
2019
;
105
(
1
):
6
21
.
doi:
10.30770/2572-1852-105.1.6
4.
Lougheed
T
.
Is it time to rethink the MCCQE Part II?
Can Med Educ J.
2016
;
7
(
1
):
e87
-
8
.
5.
Smirnova
A.
Licensing exams in Canada: a closer look at the validity of the MCCQE Part II
.
Can Med Educ J.
2022
;
13
(
v
);
23
9
.
doi:
10.36834/cmej.73894
6.
De Champlain
AF
,
Ashworth
N
,
Kain
N
,
Qin
S
,
Wiebe
D
,
Tian
F.
Does pass/fail on medical licensing exams predict future physician performance in practice? A longitudinal cohort study of Alberta physicians
.
J Med Regul.
2020
;
106
(
4
):
17
26
.
doi:
10.30770/2572-1852-106.4.17
7.
Wenghofer
E
,
Klass
D
,
Abrahamowicz
M
, et al.
Doctor scores on national qualifying examinations predict quality of care in future practice
.
Med Educ.
2009
;
43
(
12
):
1166
1173
.
doi:
10.1111/j.1365-2923.2009.03534.x
8.
Ryan
MS
,
Bishop
S
,
Browning
J
, et al.
Are scores from NBME subject examinations valid measures of knowledge acquired during clinical clerkships?
Acad Med.
2017
;
92
(
6
):
847
52
.
doi:
10.1097/ACM.0000000000001535
9.
Strowd
LC
,
Gao
H
,
Williams
DM
,
Peters
TR
,
Jackson
J.
Early preclerkship clinical skills assessments predict clerkship performance
.
Med Sci Educ.
2022
;
32
(
2
):
463
71
.
doi:
10.1007/s40670-022-01519-8
10.
Casey
PM
,
Palmer
BA
,
Thompson
GB
, et al.
Predictors of medical school clerkship performance: A multispecialty longitudinal analysis of standardized examination scores and clinical assessments
.
BMC Med Educ.
2016
;
16
:
125
.
doi:
10.1186/s12909-016-0652-y
11.
Gauer
JL
,
Jackson
J.
The association between United States Medical Licensing Examination Scores and clinical performance in medical students
.
Adv Med Educ Pract
.
2019
;
10
:
209
16
.
doi:
10.2147/AMEP.S192011
12.
Papadakis
MA
,
Arnold
GK
,
Blank
LL
,
Holmboe
ES
,
Lipner
RS.
Performance during internal medicine residency training and subsequent disciplinary action by state licensing boards
.
Ann Intern Med.
2008
;
148
(
11
):
869
76
.
doi:
10.7326/0003-4819-148-11-200806030-00009
13.
Norcini
J
,
Grabovsky
I
,
Barone
MA
,
Anderson
MB
,
Pandian
RS
,
Mechaber
AJ.
The associations between United States Medical Licensing Examination performance and outcomes of patient care
.
Acad Med.
2023
.
doi:
10.1097/ACM.0000000000005480. Epub ahead of print. PMID:
[PubMed]
.
14.
Medical Council of Canada
.
MCCQE Part I
. Accessed October 28, 2024.
15.
Medical Council of Canada
.
Virtual Delivery of the MCCQE Part II
.
Published February 18, 2021
. Accessed October 28, 2024.
16.
College of Physicians and Surgeons of Alberta
.
Annual Practice Permit Renewal
. Accessed October 28, 2024.
17.
Kolen
MJ
,
Brennan
RL.
(
2014
).
Test equating, scaling, and linking: Methods and practice (3 ed.)
.
Springer Science + Business Media
.
18.
College of Physicians and Surgeons of Alberta
.
FAQs— Albertans Complaints, Info for Complainants, Physicians
. Accessed October 28,2024.
19.
Norcini
JJ
,
Boulet
JR
,
Opalek
A
,
Dauphinee
WD.
The relationship between licensing examination performance and the outcomes of care by international medical school graduates
.
Acad Med.
2014
;
89
(
8
):
1157
62
.
doi:
10.1097/ACM.0000000000000310
20.
Tamblyn
R
,
Abrahamowicz
M
,
Dauphinee
D
, et al.
Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities
.
JAMA.
2007
;
298
(
9
):
993
1001
.
doi:
10.1001/jama.298.9.993
21.
Wenghofer
E
,
Boulet
J.
Medical Council of Canada Qualifying Examinations and performance in future practice
.
Can Med Ed J.
2022
;
13
(
4
):
53
-
61
, p.
56
.
doi:
10.36834/cmej.73770
22.
Alam
A
,
Klemensberg
J
,
Griesman
J
,
Bell
CM.
The characteristics of physicians disciplined by professional colleges in Canada
.
Open Med.
2011
;
5
(
4
):
e166
-
72
.
23.
Bismark
MM
,
Spittal
MJ
,
Gurrin
LC
,
Ward
M
,
Studdert
DM.
Identification of doctors at risk of recurrent complaints: a national study of healthcare complaints in Australia
.
BMJ Qual Saf.
2013
;
22
(
7
):
532
40
.
doi:
10.1136/bmjqs-2012-001691
24.
Spittal
MJ
,
Bismark
MM
,
Studdert
DM.
The PRONE score: an algorithm for predicting doctors’ risks of formal patient complaints using routinely collected administrative data
.
BMJ Qual Saf.
2015
;
24
:
360
-
8
.
doi:
10.1136/bmjqs-2014-003834
25.
Glover Takahashi
S
,
Nayer
M
,
St Amant
LMM.
Epidemiology of competence: a scoping review to understand the risks and supports to competence of four health professions
.
BMJ Open.
2017
;
7
(
9
):e014823.
doi:
10.1136/bmjopen-2016-014823
26.
Medical Council of Canada
.
National Registry of Physicians
. Accessed October 28, 2024. https://mcc.ca/credentials-and-services/national-registry-of-physicians.
27.
Cuddy
MM
,
Young
A
,
Gelman
A
, et al.
Exploring the relationships between USMLE performance and disciplinary action in practice: A validity study of score inferences from a licensure examination
.
Acad Med.
2017
;
92
(
12
):
1780
5
.
doi:
10.1097/ACM.0000000000001747
28.
Ali
SMH
,
Masood
HMU
,
Malik
A.
Need for a single standardized licensing & residency-entrance level exam policy in the medical education system of Pakistan
.
Pak J Med Sci.
2022
;
38
(
5
):
1083
5
.
doi:
10.12669/pjms.38.5.6464
29.
Abdel Nasser
A
,
Sharif
AF
,
Elkhamisy
FA
, et al.
Medical community perspectives regarding the Egyptian medical licensing exam: a mixed-method study
.
Cureus.
2021
;
13
(
4
):e14636.
doi:
10.7759/cureus.14636

Funding/support: This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Other disclosures: N/A

Ethics statement: This study was approved by the University of Alberta's Health Research Ethics Board Study ID: Pro00098794.

Author contributions: Study concept and design (IB, NK, NA, MM); Acquisition of data (NA, NH, KK); Analysis and/or interpretation of data (RN, IB, NA, NH, MM); Drafting of the manuscript (IB, NK, IH, HH, KK); Critical revision of the manuscript for important intellectual content (IB, NK, MM)