This report has been made available for readers of the Journal of Medical Regulation courtesy of the Pan-Canadian Physician Factors Collaboration. Copyright and all rights reserved by the Pan-Canadian Physician Factors Collaboration.

Background

The Pan-Canadian Physician Factors Collaboration is a research and development consortium that seeks to identify the specific risk and support factors that affect physician performance across a variety of contexts. A risk factor is defined as a factor that negatively impacts any facet of performance, while a support factor helps to foster, develop or improve performance (Glover Takahashi, Nayer, & St Amant, 2017). Risk factors for poor performance have been studied in the published literature, generating evidence about the individual, organizational and systems-level factors that may hinder performance. Studies have also examined support factors that may promote high-quality practices. Since the beginning of the collaboration in 2015, there has been a concerted effort to develop a strong evidence base to support the evolution of regulatory assessment programs and safe medical practices.

Risk-based prinicples have recently become a central tenet of regulation (OECD, 2012). Risk-based approaches are being applied to proactive medical regulation in several Canadian jurisdictions and a strong evidence base is needed to support this work. For example, there is interest among Canadian regulators to assess more physicians, more frequently to ensure that physicians engage in lifelong learning and maintenance of competence throughout their careers. Given limited resources and the imperative to develop assessments that foster learning, there is a growing need to ensure that physicians are routed to tailored interactions with the regulator proportionate to risk and learning needs. One strategy to accomplish this is to route physicians to appropriate assessments based on a profile that summarizes the combination of risk and support factors in a physician's practice. This approach is in line with the tenets of Right Touch Regulation, which has been the cornerstone of medical regulation in the United Kingdom over the past few years (Professionals Standards Authority, 2015).

The Pan-Canadian Physician Factors Collaboration was started by Medical Regulatory Authorities (MRAs) in Canada but other physician organizations have since joined this work. It is recognized that numerous medical stakeholders have a vested interest in how higher-risk physicians are assessed and how physicians are supported in the system as they mitigate and manage risk. Knowledge of factors that affect physician performance is relevant and important for practicing physicians, patients, medical regulatory authorities, national certifying bodies and other stakeholder groups involved in the design and delivery of healthcare. As such, a collaboration with representatives from relevant physician organizations has been convened to guide this work. This group is charged with ensuring that the program is sustainable and that the knowledge produced helps to modernize performance assessments and continuing competence systems.

Over the past few years, the collaboration has invested in developing an evidence base on risk and support factors to physician performance to achieve the following across Canada:

  • Build physician self-awareness and learning about risk and support factors

  • Create protective practice strategies

  • Select high-risk physicians for regulatory assessments

  • Promote risk-based regulatory and medico-legal practices

  • Inform physician assessment and continuing professional development programs

As of 2019, the Pan-Canadian Physician Factors Collaboration has achieved stable program status under the leadership of the Medical Council of Canada (MCC) and the Federation of Medical Regulatory Authorities of Canada (FMRAC). This formal infrastructure will facilitate on-going collaboration, resource management and knowledge translation among system partners.

Purpose

This report is a state of the science summary of what is currently known about select risk and support factors to physician performance. The purpose of this work is to compile evidence from multiple reviews into one accessible and usable document.

This is an important endeavor as the literature base is vast and it is often difficult to reach conclusions about the extent to which the evidence supports the relationship between factors and practice performance.

The current paper strives to:

  • Summarize what is known about select risk and support factors to physician performance

  • Summarize what remains unknown

  • Provide recommendations for future research

Data Sources

The data sources included in this document and the associated methodologies are described below. Systematic reviews aim to comprehensively search for all relevant articles in an area of study. A quality screen is developed to determine which articles should be included or excluded based on explicit criteria and a standardized tool is used by multiple coders to evaluate articles (Grant & Booth, 2009). Systematic reviews are the gold standard for literature reviews but they are not always feasible due to resource constraints. It is important to note that the methodologies for each of the data sources described in this synthesis do not meet the requirements of a full systematic review although systematic review methods were used to critically appraise studies.

A scoping review was commissioned to ascertain the extent to which risk and support factors were being studied in the published literature (Glover Takahashi, et al., 2017). Scoping reviews are conducted to broadly identify key concepts, theories and sources of evidence, but do not synthesize findings from individual studies (Arksey & O'Malley, 2005). Moving beyond the scoping review to evidence synthesis, the McMaster Health Forum was commissioned to conduct a rapid synthesis of factors associated with competence. They examined 17 primary studies looking at risk and support factors for quality clinical practices across three performance outcomes: complaints, disciplinary action and prescribing and testing errors (Wilson & JN, 2015). Building off of the McMaster work, a more comprehensive systematized review of factors related to complaints was conducted because the rapid review did not capture all relevant articles due to time constraints. Systematized reviews were also conducted for the following factors: being an internationally trained medical graduate, age, and exam scores (note: these were not systematic reviews).

See Table 1 for an overview of evidence syntheses developed or commissioned by MRAs and definitions for different types of review methodologies.

In addition to the above, two additional data sources contributed valuable knowledge to this report. MRA research staff led a qualitative study examining risk and supports to physician performance based on the experiential knowledge of 23 physician assessors (Kain, et al., 2018). Additionally, a systematic review exploring the relationship between clinical experience and quality of health care was uncovered as part of this work (this review was not commissioned or developed by MRAs) (Choudhry, Fletcher, & Soumerai, 2005).

All of these activities represent the current “state of the science” as it pertains to consolidated knowledge of risk and support factors to physician performance. In total, over 900 published articles were coded by MRA researchers and commissioned associates to summarize what is known about risk and support factors and reveal gaps in the evidence base.

Methods

The purpose of this report is to consolidate evidence on risk and support factors to physician performance. This is an ambitious task due to the large body of evidence that exists in this area. The individual reviews included in this report were conducted sequentially between 2015–2018. The order of the reviews was designed purposefully so that earlier reviews could inform subsequent reviews in an iterative way. For example, the scoping review served as a seminal document to understand the extent to which factors were being studied in the published literature. From this, a rapid synthesis provided early insight into the key factors that are related to different performance outcomes. Rapid syntheses are limited by time constraints so additional systematized reviews were conducted for frequently studied factors in order to consolidate the evidence base. Articles uncovered in the scoping review served as starting point for each of the systematized reviews conducted by MRA researchers and commissioned associates.

For the state of the science summary, no additional searches were conducted as the data sources listed above served as inputs to this document. To consolidate the evidence from all data sources, two researchers (WY & NT) tabulated studies that found a relationship between a particular factor and a specific performance outcome (e.g. complaints, discipline, assessment). Combing across multiple syntheses and consolidating findings by factor allowed the research team to ascertain the degree to which the evidence supported a particular factor being associated with different performance outcomes. The research team focused on risk factors frequently studied in the published literature (age, gender, exam scores, specialty, workload, solo practice, international training and experience). Each risk factor was assigned a rating: factor, probable factor, and possible factor, based on the strength of the evidence supporting each factor. This was based on the quantity and consistency of the findings throughout the literature. The quality of article was not directly considered in the rating, as the researchers were consolidating syntheses which did not have uniform quality screens.

A system was created so that factors under investigation could be rated by the strength of the evidence (Table 2).

Results

Using the rating system described, age, gender, exam scores and specialty were coded as risk factors due to the strength of the evidence. Practice experience, workload and solo practice were coded as probable risk factors given that there were fewer studies on these factors but they were still trending in a consistent direction. IMG and lack of engagement were coded as a possible risk factor given that findings are conflicted for the former and only one study exists for the latter.

Table 3 provides a high-level summary of findings. Tables 4, 5 and 6 provide detailed findings as well as implications of the results. References for the detailed findings can be found in  Appendix 2.

Discussion

To date, knowledge of risk and support factors that impact physician performance has never been consolidated in a comprehensive and usable way. It is envisioned that such knowledge will benefit physicians as well as physician organizations in the healthcare system. Physicians will be provided with knowledge to proactively mitigate risks in their practice before they arise. It is important to note that some risk factors are not modifiable and physicians should be encouraged to consider the information in the context of their own practices. MRAs will use knowledge of risk factors to triage physicians for assessment so that resources are used more efficiently to support risk-based regulation. National certifying colleges responsible for setting standards for training, certification and lifelong learning will use knowledge of factors to shape personal learning plans and develop programs that help physicians use supports to mitigate risk. Other stakeholders in the system, such as the Medical Council of Canada and the Canadian Medical Protective Association, will also use knowledge of factors to understand, support and protect physician practices. All of the above endeavors are geared towards promoting high-quality practices and an effective continuing competence system.

Several MRAs across Canada are already using consolidated knowledge of factors to inform regulatory processes and programs. British Columbia, Alberta, Ontario, Quebec and Nova Scotia are using knowledge of factors to select physicians for assessment. Quebec has been risk-prioritizing physicians for many years by using their in-house data to understand the associations between various demographic and practice factors and assessment outcomes. Other jurisdictions are begining to use the knowledge gained through Quebec's experience along with the evidence base to inform selection processes. Nova Scotia, Ontario and Alberta are also using knowledge of factors to develop assessment programming to inform physicians of risks and supports to practice. For example, Ontario is developing a self-directed online interactive tool that will provide physicians with an individualized practice profile of risk and support factors. A report will be made available through this process, which will provide physicians with mitigating strategies and resources available for practice improvement.

It should be highlighted that although this report consolidates evidence across markers of performance (e.g. complaints, discipline, assessment, etc.), it is always important to consider “at risk of what?”. Physician performance is a multi-dimensional construct and the quality of care delivered in one domain does not presuppose quality in another. Characteristics that are associated with physicians who have been disciplined may be different than those who have an unsuccessful assessment outcome. Both disciplinary action and assessment results are a measure of different aspects of poor performance. When applying the evidence in this summary, the context, purpose and what facet of care the program is measuring, assessing or facilitating needs to be considered.

Strengths

One of the key strengths of this work is that a comprehensive state of the science summary has never been conducted in this area to take stock of what is known. This work is valuable as evidence from multiple reviews is consolidated in a usable way to guide policy and program development. In the absence of such work, it is difficult to ascertain the strength of the evidence across multiple data sources. This work also represents a significant contribution in that considerable resources were dedicated to the individual inputs included in this report (i.e. numerous researchers and academic partners contributed to each of the individual syntheses). Through the scoping review, a comprehensive reference list of 754 studies that examined risk and support factors to physician competence was generated and will be used as a starting point to synthesize evidence about factors not covered in this report.

While the reviews included in the report do not meet the criteria for systematic review (with the exception of one), systematized methodology was used for each of the inputs to ensure rigor. Our understanding of select factors was enhanced by the ability to study their impact across a variety of performance outcomes (e.g. prescribing, assessment, complaints) as well as across various samples and settings due to the heterogenous nature of individual studies. In total, over 900 studies were coded as part of this analysis and we were able to ascertain the strength of the evidence for some well-studied risk factors as well as reveal gaps in the literature base. This consolidated work can now be shared with national and international audiences to inform risk-based approaches to regulation and continuing competence systems.

Limitations

Limitations also exist for the work described in this report. The methodology does not meet the rigor of a full systematic review, which means that some articles may have been missed, quality screens may or may not have been applied, depending on the review, and multiple coders were not always used. While this work represents a significant contribution, the research team was only able to synthesize evidence on select risk factors due to resource constraints. As such, this was not an exploratory exercise to catalogue all the possible factors, but rather, a confirmatory exercise to consolidate and summarize what is known. The impact of risk factors was considered for the performance outcomes most frequently studied in the published literature and does not represent all possible performance outcomes. As such, more work is needed to understand risk factors above and beyond what has been included in this report. For example, while the scoping review uncovered published articles on support factors, we have not taken an in-depth look at these studies and know relatively little compared to our knowledge of risk factors.

While the evidence base on risk and support factors to performance is growing steadily, there remains a scarcity of information on why identified factors may hinder or support practice. This dearth can be largely explained by the methodological approaches used to study risk and support factors to date; most of the published literature utilizes epidemiological approaches on epidemiological data, which does not shed light on why a factor might be a factor and typically only reveal correlations, not causal relationships. Finally, it is unorthodox to conduct a state of the science summary that includes syntheses that are not yet available in the public domain. We strategically decided to prioritize the current high-level synthesized findings of multiple reviews to facilitate sharing the knowledge with administrators and program developers in a timely manner.

Future Research Agenda

The evidence base will continue to grow as more factors research is published. A national research agenda will be developed and formalized under the direction of the Steering Committee and MCC/FMRAC to address gaps inclusive of the following:

  • Due to methodological constraints and availability of data, some factors are easier to study than others, which may contribute to why they are strong predictors of performance in the published literature (e.g. gender). There are likely risk and support factors that are strongly predictive of practice performance but are more difficult to measure and study (e.g. lack of engagement, patient centeredness). These important factors will need to be studied using different methodological approaches, such as qualitative research, which does not rely on administrative data.

  • Other factors not summarized in this report but identified in the literature include: lack of board or specialty certification, practice location, previous complaints or medico-legal events, practicing in a hospital environment and engaging in certain types of continuing professional development. Future work should examine evidence for these factors as well as study how organizational and system levels factors affect performance.

  • Qualitative methodologies will also help us to understand why a factor is a factor and not just whether or not it is associated with practice performance.

  • For factors where there is a lot of evidence (e.g. age, gender), future studies can stratify groups to garner more nuanced understanding. For example, stratifying by gender will allow researchers to understand the specific risk factors associated with being male. Stratifying by age cohorts will contribute to understanding factors pertinent to early, middle, and late career practice.

  • MRAs are well positioned to make a significant contribution to the existing evidence base due to the wealth of performance data that exists in these organizations.

  • Factors associated with performance outcomes not included in this review will need to be studied in future work (e.g. billing data, patient outcomes).

  • Compared to knowledge of risk factors, we know relatively little about factors that support high-quality practices. This will be a key area for future work as system partners focus on preventive strategies to facilitate the development of safe practices.

  • New directions for future research should focus on trends in health care inclusive of the role of context in performance, how to assess team-based care and the role of support factors in promoting high-quality practices.

Key Takeaways

To date, evidence of risk and supports to performance has never been consolidated in such a comprehensive way. The following represents key takeaways from this work:

  • The evidence base on risk and supports to physician practice will continuously evolve as practice changes and new knowledge is added to the system. It is the intention of the national collaboration to continue to update the evidence base so it can be iteratively applied to regulatory programming to ensure relevance.

  • It is always important to consider “at risk of what?” as risk factors differ across performance outcomes.

  • Audiences will need to bear in mind that the methodologies used in individual studies within each review examine associations between variables. This is not the same as examining the relationship between cause and effect among variables. As such, those with high-risk profiles will not necessarily perform more poorly, although they are more likely to.

  • Future national work will focus on developing the necessary infrastructure to share factors knowledge across MRAs, physician organizations and academic partners. The appropriate knowledge translation infrastructure is to be built under the guidance of the collaboration and MCC/FMRAC.

  • The knowledge generated in this document will need to be applied to processes and programs in the system to measure the added benefit to this approach (e.g., are triage tools developed to select at-risk physicians effective? Does providing physicians with knowledge of risk and support factors help to protect practices?).

Acknowledgments

This report was created on behalf of the Pan-Canadian Physician Factors Collaboration under the direction of the Co-chairs, Daniel Faulkner of the College of Physicians and Surgeons of Ontario and Dr. Karen Mazurek of the College of Physicians and Surgeons of Alberta. We thank Dr. Nigel Ashworth of the College of Physicians and Surgeons of Alberta for his contribution to the risk rating framework.

The following organizations are involved in the Canadian Physician Factors Collaboration:

  • College of Physicians and Surgeons of Alberta (CPSA)

  • College of Physicians and Surgeons of British Columbia (CPSBC)

  • College of Physicians and Surgeons of Manitoba (CPSM)

  • College of Physicians and Surgeons of Nova Scotia (CPSNS)

  • College of Physicians and Surgeons of Ontario (CPSO)

  • College des Medecins du Quebec (CMQ)

  • College of Family Physicians Canada (CFPC)

  • Canadian Medical Protective Association (CMPA)

  • Federation of Medical Regulatory Authorities of Canada (FMRAC)

  • Medical Council of Canada (MCC)

  • Royal College of Physicians and Surgeons of Canada (RCPSC)

General References

General References
Arksey
,
H.
, &
O'Malley
,
L.
(
2005
).
Scoping studies: Towards a methodological framework
.
International Journal of Social Research Methodology: Theory and Practice
,
8
(
1
),
19
32
.
Choudhry
,
N. K.
,
Fletcher
,
R. H.
, &
Soumerai
,
S. B.
(
2005
).
Systematic Review: The Relationship between Clinical Experience and Quality of Health Care
.
Annals of Internal Medicine
,
142
(
4
),
260
273
.
Glover Takahashi
,
S.
,
Nayer
,
M.
, &
St Amant
,
L. M. M.
(
2017
).
Epidemiology of competence: A scoping review to understand the risks and supports to competence of four health professions
.
BMJ Open
,
7
(
9
),
1
12
.
Grant
,
M. J.
, &
Booth
,
A.
(
2009
).
A typology of reviews: An analysis of 14 review types and associated methodologies
.
Health Information and Libraries Journal
,
26
(
2
),
91
108
.
Kain
,
N.
,
Hodwitz
,
K.
,
Yen
,
W.
&
Ashworth
,
N.
(
2018
).
Experiential Knowledge of Risk and Support Factors for Physician Performance in Canada: A Qualitative Study
.
BMJ Open
.
Volume 9
(
2
).
OECD
.
(
2012
).
Recommendation of the Council on Regulatory Policy and Governance
.
The OECD Regulatory Policy Committee
.
Wilson
,
M. G.
, &
JN
,
R. H. L.
(
2015
).
Rapid Synthesis: Identifying Risk and Protective Factors for Quality Clinical Practice
.
McMaster Health Forum
.

Appendix 1 Detailed Information of Data Sources

Appendix 2 References for Detailed Findings

  • RISK FACTORS

    16

  • Age

    16

  •   Complaints

    16

  •   Physical Performance

    16

  •   Cognitive Performance

    16

  •   Psychological Wellness

    16

  •   Clinical Knowledge or Performance

    17

  •   Patient Safety

    17

  •   Medico-Legal

    18

  • Gender

    18

  •   Complaints

    18

  •   Discipline

    19

  •   Prescribing and Ordering Tests

    19

  • Exam Scores

    19

  • Specialty

    19

  •   Complaints

    19

  •   Discipline

    20

  • PROBABLE RISK FACTORS

    20

  • Practice Experience

    20

  •   Discipline

    20

  •   Other Performance Outcomes

    20

  • Workload

    20

  • Solo Practice

    20

  • Lack of Engagement/Patient Centeredness

    20

  • POSSIBLE RISK FACTORS

    20

  • IMG

    20

  •   Educational Performance

    20

  •   Complaints

    21

  •   Discipline

    21

  •   Other Performance Outcomes

    21

RISK FACTORS

RISK FACTORS
Age: Complaints
Bismark
,
M. M.
,
Spittal
,
M. J.
,
Gurrin
,
L. C.
,
Ward
,
M.
, &
Studdert
,
D. M.
(
2013
).
Identification of doctors at at risk of recurrent complaints: a national study of healthcare complaints in Australia
.
BMJ Quality & Safety
,
22
(
7
),
532
540
.
Enbom
,
J. A.
, &
Thomas
,
C. D.
(
1997
).
Evaluation of sexual misconduct complaints: The Oregon Board of Medical Examiners, 1991 to 1995
.
American Journal of Obstetrics and Gynecology
,
176
(
6
),
1340
1348
.
Hickson
,
G. B.
,
Federspiel
,
C. F.
,
Pichert
,
J. W.
,
Miller
,
C. S.
,
Gauld-Jaeger
,
J.
, &
Bost
,
P.
(
2002
).
Patient complaints and malpractice risk
.
JAMA: The Journal of the American Medical Association
,
287
(
22
),
2951
2957
.
Murad
,
M. H.
,
Gjerde
,
C. L.
,
Bobula
,
J.
,
Ostrov
,
M.
, &
Murad
,
M. S.
(
2009
).
Gender and patient complaints: Are they related?
Quality in Primary Care
,
17
(
5
),
351
357
.
Spittal
,
M. J.
,
Bismark
,
M. M.
, &
Studdert
,
D. M.
(
2015
).
The PRONE score: an algorithm for predicting doctors' risks of formal patient complaints using routinely collected administrative data
.
BMJ Quality & Safety
,
24
(
6
),
360
8
.
Stelfox
,
H. T.
,
Gandhi
,
T. K.
,
Orav
,
E. J.
, &
Gustafson
,
M. L.
(
2005
).
The relation of patient satisfaction with complaints against physicians and malpractice lawsuits
.
The American Journal of Medicine
,
118
(
10
),
1126
33
.
Tibble
,
H. M.
,
Broughton
,
N. S.
,
Studdert
,
D. M.
,
Spittal
,
M. J.
,
Hill
,
N.
,
Morris
,
J. M.
, &
Bismark
,
M. M.
(
2017
).
Why do surgeons receive more complaints than their physician peers?
ANZ Journal of Surgery
,
87
(
9
).
Wenghofer
,
E. F.
,
Campbell
,
C.
,
Marlow
,
B.
,
Kam
,
S. M.
,
Carter
,
L.
, &
McCauley
,
W.
(
2015
).
The effect of continuing professional development on public complaints: a case-control study
.
Medical Education
,
49
(
3
),
264
75
.
Age: Physical Performance
Bieliauskas
LA
,
Langenecker
S
,
Graver
C
,
Lee
HJ
,
O'Neill
J
,
Greenfield
LJ.
(
2008
).
Cognitive Changes and Retirement among Senior Surgeons (CCRASS): Results from the CCRASS Study
.
J Am Coll Surg
,
207
(
1
):
69
79
.
Boom-Saad
Z
,
Langenecker
SA
,
Bieliauskas
LA
,
et al
. (
2008
).
Surgeons outperform normative controls on neuropsychologic tests, but age-related decay of skills persists
.
The American Journal of Surgery
,
195
(
2
):
205
209
.
Drag
LL
,
Bieliauskas
LA
,
Langenecker
SA
,
Greenfield
LJ.
(
2010
).
Cognitive Functioning, Retirement Status, and Age: Results from the Cognitive Changes and Retirement among Senior Surgeons Study
.
J Am Coll Surg
,
211
:
303
307
.
Harms
BA
,
Heise
CP
,
Gould
JC
,
Starling
JR.
(
2005
).
A 25-year single institution analysis of health, practice, and fate of general surgeons
.
Annals of Surgery
,
242
(
4
):
520
526
.
Kataria
N
,
Brown
N
,
McAvoy
P
,
Majeed
A
,
Rhodes
M.
(
2014
).
A retrospective study of cognitive function in doctors and dentists with suspected performance problems: an unsuspected but significant concern
.
J R Soc Med
,
5
(
5
):
1
9
.
Onyura
B
,
Bohnen
J
,
Wasylenki
D
,
et al
. (
2015
).
Reimagining the Self at Late-Career Transitions: How Identity Threat Influences Academic Physicians' Retirement Considerations
.
Academic Medicine
,
90
(
6
):
794
801
.
Perry
W
,
Crean
RD.
(
2005
).
A retrospective review of the neuro-psychological test performance of physicians referred for medical infractions
.
Archives of Clinical Neuropsychology
,
20
:
161
170
.
Wallace
MS
,
Ashman
MN
,
Matjasko
MJ.
(
1994
).
Hearing acuity of anesthesiologists and alarm detection
.
Anesthesiology
,
81
:
13
28
.
Age: Cognitive Performance
Bieliauskas
LA
,
Langenecker
S
,
Graver
C
,
Lee
HJ
,
O'Neill
J
,
Greenfield
LJ.
(
2008
).
Cognitive Changes and Retirement among Senior Surgeons (CCRASS): Results from the CCRASS Study
.
J Am Coll Surg
,
207
(
1
):
69
79
.
Boom-Saad
Z
,
Langenecker
SA
,
Bieliauskas
LA
,
et al
. (
2008
).
Surgeons outperform normative controls on neuropsychologic tests, but age-related decay of skills persists
.
The American Journal of Surgery
,
195
:
205
209
.
Drag
LL
,
Bieliauskas
LA
,
Langenecker
SA
,
Greenfield
LJ.
(
2010
).
Cognitive Functioning, Retirement Status, and Age: Results from the Cognitive Changes and Retirement among Senior Surgeons Study
.
J Am Coll Surg
,
211
:
303
307
.
Goldberg
R
,
Thomas
H
,
Penner
L.
(
2011
).
Issues of Concern to Emergency Physicians in Pre-Retirement Years: A Survey
.
The Journal of Emergency Medicine
,
40
(
6
):
706
713
.
Kataria
N
,
Brown
N
,
McAvoy
P
,
Majeed
A
,
Rhodes
M.
(
2014
).
A retrospective study of cognitive function in doctors and dentists with suspected performance problems: an unsuspected but significant concern
.
J R Soc Med
,
5
(
5
):
1
9
.
Lee
HJ
,
Drag
LL
,
Bieliauskas
LA
,
et al
. (
2009
).
Results from the Cognitive Changes and Retirement among Senior Surgeons Self-Report Survey
.
J Am Coll Surg
,
209
(
5
):
668
671
.
Onyura
B
,
Bohnen
J
,
Wasylenki
D
,
et al
. (
2015
).
Reimagining the Self at Late-Career Transitions: How Identity Threat Influences Academic Physicians' Retirement Considerations
.
Academic Medicine
,
90
(
6
):
794
801
.
Peisah
C
,
Gautam
M
,
Goldstein
MZ.
(
2009
).
Medical masters: A pilot study of adaptive ageing in physicians
.
Australian Journal on Ageing
,
28
(
3
):
134
138
.
Perry
W
,
Crean
RD.
(
2005
).
A retrospective review of the neuro-psychological test performance of physicians referred for medical infractions
.
Archives of Clinical Neuropsychology
,
20
(
2
):
161
170
.
Turnbull
J
,
Carbotte
R
,
Hanna
E
,
et al
. (
2000
).
Cognitive difficulty in physicians
.
Academic Medicine
,
75
(
2
):
177
181
.
Turnbull
J
,
Cunnington
J
,
Unsal
A
,
Norman
G
,
Ferguson
B.
(
2006
).
Competence and Cognitive Difficulty in Physicians: A Follow-up Study
.
Academic Medicine
,
81
(
10
):
915
918
.
Age: Psychological Wellness
Goldberg
R
,
Thomas
H
,
Penner
L.
(
2011
).
Issues of Concern to Emergency Physicians in Pre-Retirement Years: A Survey
.
The Journal of Emergency Medicine
,
40
(
6
):
706
713
.
Harms
BA
,
Heise
CP
,
Gould
JC
,
Starling
JR.
(
2005
).
A 25-year single institution analysis of health, practice, and fate of general surgeons
.
Annals of Surgery
,
242
(
4
):
520
526
.
Hull
SK
,
DiLalla
LF
,
Dorsey
JK.
(
2008
).
Prevalence of health-related behaviors among physicians and medical trainees
.
Academic Psychiatry
,
32
(
1
):
31
38
.
Peisah
C
,
Gautam
M
,
Goldstein
MZ.
(
2009
).
Medical masters: A pilot study of adaptive ageing in physicians
.
Australian Journal on Ageing
,
28
(
3
):
134
138
.
Ross
LK
,
Fox
PJ
,
Gould
D
,
Kohatsu
ND.
(
2004
).
Characteristics Associated With Physician Discipline: A Case-Control Study
.
Archives of Internal Medicine
,
164
(
6
):
653
658
.
Age: Clinical Knowledge or Performance
Acton
RT
,
Barton
JC
,
Casebeer
L
,
Talley
L.
(
2002
).
Survey of physician knowledge about hemochromatosis
.
Genetics in Medicine
,
4
(
3
):
136
141
.
Al-Maniri
AA
,
Al-Rawas
OA
,
Al-Ajmi
F
,
De Costa
A
,
Eriksson
B
,
Diwan
VK.
(
2008
).
Tuberculosis suspicion and knowledge among private and public general practitioners: Questionnaire Based Study in Oman
.
BMC Public Health
,
8
:
177
.
Beam
CA
,
Conant
EF
,
Sickles
EA.
(
2003
).
Association of volume and volume-independent factors with accuracy in screening mammogram interpretation
.
Journal of the National Cancer Institute
,
95
(
4
):
282
290
.
Caulford
PG
,
Lamb
SB
,
Kaigas
TB
,
Hanna
E
,
Norman
GR
,
Davis
DA.
(
1994
).
Physician incompetence: specific problems and predictors
.
Academic Medicine
,
69
(
10 Suppl
):
S16
18
.
Charap
MH
,
Levin
RI
,
Weinglass
J.
(
1985
).
Physician choices in the treatment of angina pectoris
.
American Journal of Medicine
,
79
(
4
):
461
466
.
Chauvel
N
,
Le Vaillant
M
,
Pelletier-Fleury
N.
(
2013
).
Variation in HbA1c prescription for patients with diabetes in French general practice: an observational study prior to the implementation of a P4P programme
.
Eur J Public Health
,
23
(
1
):
61
66
.
Epstein
S
,
Gonzales
J.
(
2001
).
Are psychiatrists' characteristics related to how they care for depression in the medically ill? Results from a national case-vignette survey
.
Psychosomatics
,
42
:
482
489
.
Frost-Pineda
K
,
VanSusteren
T
,
Gold
MS.
(
2004
).
Are physicians and medical students prepared to educate patients about alcohol consumption?
Journal of Addictive Diseases
,
23
(
2
):
1
13
.
Gerstein
HC
,
Reddy
SS
,
Dawson
KG
,
Yale
JF
,
Shannon
S
,
Norman
G.
(
1999
).
A controlled evaluation of a national continuing medical education programme designed to improve family physicians' implementation of diabetes-specific clinical practice guidelines
.
Diabetic Medicine
,
16
(
11
):
964
969
.
Goulet
F
,
Gagnon
RJ
,
Desrosiers
G
,
Jacques
A
,
Sindon
A.
(
1998
).
Participation in CME activities
.
Canadian Family Physician
,
44
:
541
548
.
Grace
ES
,
Korinek
EJ
,
Weitzel
LB
,
Wentz
DK.
(
2011
).
Physicians Reentering Clinical Practice: Characteristics and Clinical Abilities
.
Journal of Continuing Education in the Health Professions
,
31
(
1
):
49
55
.
Grace
ES
,
Wenghofer
EF
,
Korniek
EJ.
(
2014
).
Predictors of Physician Performance on Competence Assessment: Findings From CPEP, the Center for Personalized Education for Physicians
.
Academic Medicine
,
89
(
6
):
912
919
.
Haque
AS
,
Zubairi
AB
,
Shiraz
A
,
et al
. (
2007
).
Asthma knowledge and approach among Pakistani family physicians and the impact of an educational programme
.
International Journal of Tuberculosis and Lung Disease
,
11
(
11
):
1260
1265
.
Hofman
KJ
,
Tambor
ES
,
Chase
GA
,
Geller
G
,
Faden
RR
,
Holtzman
NA.
(
1993
).
Physicians' knowledge of genetics and genetic tests
.
Academic Medicine
,
68
(
8
):
625
632
.
Lewis
CE
,
Freeman
HE
,
Kaplan
SH
,
Corey
CR.
(
1986
).
The impact of a program to enhance the competencies of primary care physicians in caring for patients with AIDS
.
Journal of General Internal Medicine
,
1
(
5
):
287
294
.
Lipner
R
,
Song
H
,
Biester
T
,
Rhodes
R.
(
2011
).
Factors That Influence General Internists' and Surgeons' Performance on Maintenance of Certification Exams
.
Academic Medicine
,
86
(
1
):
53
58
.
McAuley
RG
,
Paul
WM
,
Morrison
GH
,
Beckett
RF
,
Goldsmith
CH.
(
1990
).
Five-year results of the peer assessment program of the College of Physicians and Surgeons of Ontario
.
Canadian Medical Association Journal
,
143
(
11
):
1193
1199
.
Meskauskas
JA
,
Webster
GD.
(
1975
).
The American Board of Internal Medicine Recertification Examination: Process and Results
.
Annals of Internal Medicine
,
82
(
4
):
577
Mujtaba
SH
,
Ashraf
T
,
Anjum
Q.
(
2013
).
Improving general practitioners' knowledge regarding blood pressure measurement in selected cities of Pakistan through workshop
.
Asia-Pacific Journal of Public Health
,
25
(
1
):
84
91
.
Norcini
JJ
,
Fletcher
SW
,
Quimby
BB
,
Shea
JA.
(
1985
).
Performance of women candidates on the American Board of Internal Medicine Certifying Examination, 1973–1982
.
Annals of Internal Medicine
,
102
(
1
):
115
118
.
Norman
GR
,
Davis
DA
,
Caulford
P
,
Kaigas
T
,
Hanna
E
,
Lamb
S.
(
1993
).
Competency Assessment of Primary Care Physicians as Part of a Peer Review Program
.
JAMA: The Journal of the American Medical Association
,
270
(
9
):
1046
1051
.
Norman
GR
,
Wenghofer
E
,
Klass
D.
(
2008
).
Predicting doctor performance outcomes of curriculum interventions: problem-based learning and continuing competence
.
Medical Education
,
42
:
794
799
.
Norton
PG
,
Dunn
EV
,
Soberman
L.
(
1997
).
What Factors Affect Quality of Care?
Canadian Family Physician
,
43
:
1739
1744
.
Pentzek
M
,
Abholz
HH
,
Ostapczuk
M
,
Altiner
A
,
Wollny
A
,
Fuchs
A.
(
2009
).
Dementia knowledge among general practitioners: first results and psychometric properties of a new instrument
.
International Psychogeriatrics
,
21
(
6
):
1105
1115
.
Sample
L
,
Laduca
T
,
Leung
C
,
et al
. (
2001
).
Comparing patient-management skills of referred physicians and non-referred physicians on a computer-based case-simulation examination
.
Academic Medicine
,
76
(
10 Suppl
):
S24
.
Savatsky
PD
,
Haitz
MC
,
Sterns
NS.
(
1981
).
Patterns of continuing medical education: a generation unit analysis of physicians in three community hospitals
.
Social Science & Medicine - Part A, Medical Sociology
,
15
(
5
):
665
672
.
Schroen
A.
(
2000
).
Beliefs among pulmonologists and thoracic surgeons in the therapeutic approach to non-small cell lung cancer
.
Chest
,
118
(
1
):
129
137
.
Sharma
R
,
Kostis
WJ
,
Wilson
AC
,
et al
. (
2008
).
Questionable hospital chart documentation practices by physicians
.
Journal of General Internal Medicine
,
23
(
11
):
1865
1870
.
Wenghofer
E
,
Klass
D
,
Abrahamowicz
M
,
et al
. (
2009
).
Doctor scores on national qualifying examinations predict quality of care in future practice
.
Medical Education
,
43
(
12
):
1166
1173
.
Xierali
IM
,
Rinaldo
JC
,
Green
LA
,
et al
. (
2011
).
Family physician participation in maintenance of certification
.
Annals of Family Medicine
,
9
(
3
):
203
210
.
Zakroyeva
A
,
Goldberg
D
,
Gask
L
,
Leese
M.
(
2008
).
Training Russian family physicians in mental health skills
.
European Journal of General Practice
,
14
(
1
):
19
22
.
Age: Patient Safety
Brekke
M
,
Rognstad
S
,
Straand
J
,
et al
. (
2008
).
Pharmacologically inappropriate prescriptions for elderly patients in general practice: How common? Baseline data from The Prescription Peer Academic Detailing (Rx-PAD) study
.
Scand J Prim Health Care
,
26
(
2
):
80
85
.
Charap
MH
,
Levin
RI
,
Weinglass
J.
(
1985
).
Physician choices in the treatment of angina pectoris
.
American Journal of Medicine
,
79
(
4
):
461
466
.
Chauvel
N
,
Le Vaillant
M
,
Pelletier-Fleury
N.
(
2013
).
Variation in HbA1c prescription for patients with diabetes in French general practice: an observational study prior to the implementation of a P4P programme
.
Eur J Public Health
,
23
(
1
):
61
66
.
Dhalla
I.
(
2002
).
Inappropriate prescribing before and after nursing home admission
.
J Am Geriatr Soc
,
50
(
6
):
995
1000
.
Duclos
A
,
Peix
J-L
,
Colin
C
,
et al
. (
2012
).
Influence of experience on performance of individual surgeons in thyroid surgery: prospective cross sectional multicentre study
.
British Medical Journal
,
344
.
Epstein
S
,
Gonzales
J.
(
2001
).
Are psychiatrists' characteristics related to how they care for depression in the medically ill? Results from a national case-vignette survey
.
Psychosomatics
,
42
:
482
489
.
Grace
ES
,
Wenghofer
EF
,
Korniek
EJ.
(
2014
).
Predictors of Physician Performance on Competence Assessment: Findings From CPEP, the Center for Personalized Education for Physicians
.
Academic Medicine
,
89
(
6
):
912
919
.
Hartz
AJ
,
Kuhn
EM
,
Pulido
J.
(
1999
).
Prestige of training programs and experience of bypass surgeons as factors in adjusted patient mortality rates
.
Medical Care
,
37
(
1
):
93
103
.
McAuley
RG
,
Henderson
HW.
(
1984
).
Results of the peer assessment program of the College of Physicians and Surgeons of Ontario
.
Canadian Medical Association Journal
,
131
(
6
):
557
561
.
Norcini
JJ
,
Kimball
HR
,
Lipner
RS.
(
2000
).
Certification and specialization: do they matter in the outcome of acute myocardial infarction?
Academic Medicine
,
75
(
12
):
1193
1198
.
Norcini
JJ
,
Lipner
RS
,
Kimball
HR.
(
2002
).
Certifying examination performance and patient outcomes following acute myocardial infarction
.
Medical Education
,
36
(
9
):
853
859
.
O'Neill
L
,
Lanska
DJ
,
Hartz
AJ.
(
2000
).
Surgeon characteristics associated with mortality and morbidity following carotid endarterectomy
.
Neorology
,
55
(
September
):
773
781
.
Waljee
JF
,
Greenfield
LJ
,
Dimick
JB
,
Birkmeyer
JD.
(
2006
).
Surgeon Age and Operative Mortality in the United States
.
Annals of Surgery
,
244
(
3
):
353
362
.
Age: Medico-Legal
Bismark
MM
,
Spittal
MJ
,
Studdert
DM.
(
2011
).
Prevalence and characteristics of complaint-prone doctors in private practice in Victoria
.
Med J Aust
,
195
(
1
):
25
28
.
Caulford
PG
,
Lamb
SB
,
Kaigas
TB
,
Hanna
E
,
Norman
GR
,
Davis
DA.
(
1994
).
Physician incompetence: specific problems and predictors
.
Academic Medicine
,
69
(
10 Suppl
):
S16
18
.
Donaldson
LJ
,
Panesar
SS
,
McAvoy
PA
,
Scarrott
DM.
(
2014
).
Identification of poor performance in a national medical work-force over 11 years: an observational study
.
BMJ Qual Saf
,
23
(
2
):
147
152
.
Grace
ES
,
Wenghofer
EF
,
Korniek
EJ.
(
2014
).
Predictors of Physician Performance on Competence Assessment: Findings From CPEP, the Center for Personalized Education for Physicians
.
Academic Medicine
,
89
(
6
):
912
919
.
Harms
BA
,
Heise
CP
,
Gould
JC
,
Starling
JR.
(
2005
).
A 25-year single institution analysis of health, practice, and fate of general surgeons
.
Annals of Surgery
,
242
(
4
):
520
526
.
Kataria
N
,
Brown
N
,
McAvoy
P
,
Majeed
A
,
Rhodes
M.
(
2014
).
A retrospective study of cognitive function in doctors and dentists with suspected performance problems: an unsuspected but significant concern
.
J R Soc Med
,
5
(
5
):
1
9
.
Khaliq
AA
,
Dimassi
H
,
Huang
C-Y
,
Narine
L
,
Smego
RA.
(
2005
).
Disciplinary action against physicians: Who is likely to get disciplined?
The American Journal of Medicine
,
118
(
7
):
773
777
.
Norman
GR
,
Davis
DA
,
Caulford
P
,
Kaigas
T
,
Hanna
E
,
Lamb
S.
(
1993
).
Competency Assessment of Primary Care Physicians as Part of a Peer Review Program
.
JAMA: The Journal of the American Medical Association
,
270
(
9
):
1046
1051
.
Perry
W
,
Crean
RD.
(
2005
).
A retrospective review of the neuro-psychological test performance of physicians referred for medical infractions
.
Archives of Clinical Neuropsychology
,
20
(
2
):
161
170
.
Ross
LK
,
Fox
PJ
,
Gould
D
,
Kohatsu
ND.
(
2004
).
Characteristics Associated With Physician Discipline: A Case-Control Study
.
Archives of Internal Medicine
,
164
(
6
):
653
658
.
Sample
L
,
Laduca
T
,
Leung
C
,
et al
. (
2001
).
Comparing patient-management skills of referred physicians and non-referred physicians on a computer-based case-simulation examination
.
Academic Medicine
,
76
(
10 Suppl
):
S24
6
.
Taragin
MI
,
Wilczek
AP
,
Karns
ME
,
Trout
R
,
Carson
JL.
(
1992
).
Physician demographics and the risk of medical malpractice
.
American Journal of Medicine
,
93
(
5
):
537
542
.
Tibble
,
H. M.
,
Broughton
,
N. S.
,
Studdert
,
D. M.
,
Spittal
,
M. J.
,
Hill
,
N.
,
Morris
,
J. M.
, &
Bismark
,
M. M.
(
2017
).
Why do surgeons receive more complaints than their physician peers?
ANZ Journal of Surgery
,
87
(
9
).
Turnbull
J
,
Cunnington
J
,
Unsal
A
,
Norman
G
,
Ferguson
B.
(
2006
).
Competence and Cognitive Difficulty in Physicians: A Follow-up Study
.
Academic Medicine
,
81
(
10
):
915
918
.
Wenghofer
EF
,
Campbell
C
,
Marlow
B
,
Kam
SM
,
Carter
L
,
McCauley
W.
(
2015
).
The effect of continuing professional development on public complaints: a case-control study
.
Medical Education
,
49
:
264
275
.
Wickersham
P
,
Morrison
J.
(
1998
).
Physicians Disciplined by a State Medical Board
.
JAMA: The Journal of the American Medical Association
,
279
(
23
):
1889
1893
.
Gender: Complaints
Bismark
,
M. M.
,
Spittal
,
M. J.
,
Gurrin
,
L. C.
,
Ward
,
M.
, &
Studdert
,
D. M.
(
2013
).
Identification of doctors at risk of recurrent complaints: a national study of healthcare complaints in Australia
.
BMJ Quality & Safety
,
22
(
7
),
532
540
.
Bismark
,
M. M.
,
Spittal
,
M. J.
, &
Studdert
,
D. M.
(
2011
).
Prevalence and characteristics of complaint-prone doctors in private practice in Victoria
.
The Medical Journal of Australia
,
195
(
1
),
25
8
.
Elkin
,
K.
,
Spittal
,
M.
, &
Studdert
,
D.
(
2012
).
Risks of complaints and adverse disciplinary findings against international medical graduates in Victoria and Western Australia
.
Medical Journal
,
197
,
448
452
.
Enbom
,
J. A.
, &
Thomas
,
C. D.
(
1997
).
Evaluation of sexual misconduct complaints: The Oregon Board of Medical Examiners, 1991 to 1995
.
American Journal of Obstetrics and Gynecology
,
176
(
6
),
1340
1348
.
Hickson
,
G. B.
,
Federspiel
,
C. F.
,
Blackford
,
J.
,
Pichert
,
J. W.
,
Gaska
,
W.
,
Merrigan
,
M. W.
, &
Miller
,
C. S.
(
2007
).
Patient Complaints and Malpractice Risk in a Regional Healthcare Center
.
Southern Medical Journal
,
100
(
8
),
791
796
.
Hickson
,
G. B.
,
Federspiel
,
C. F.
,
Pichert
,
J. W.
,
Miller
,
C. S.
,
Gauld-Jaeger
,
J.
, &
Bost
,
P.
(
2002
).
Patient complaints and malpractice risk
.
JAMA: The Journal of the American Medical Association
,
287
(
22
),
2951
2957
.
Kain
,
N.
,
Hodwitz
,
K.
,
Yen
,
W.
, &
Ashworth
,
N.
(
2018
).
Experiential knowledge of physician risk & support factors
.
BMJ Open
(
in press
).
Murad
,
M. H.
,
Gjerde
,
C. L.
,
Bobula
,
J.
,
Ostrov
,
M.
, &
Murad
,
M. S.
(
2009
).
Gender and patient complaints: Are they related?
Quality in Primary Care
,
17
(
5
),
351
357
.
Spittal
,
M. J.
,
Bismark
,
M. M.
, &
Studdert
,
D. M.
(
2015
).
The PRONE score: an algorithm for predicting doctors' risks of formal patient complaints using routinely collected administrative data
.
BMJ Quality & Safety
,
24
(
6
),
360
8
.
Stelfox
,
H. T.
,
Gandhi
,
T. K.
,
Orav
,
E. J.
, &
Gustafson
,
M. L.
(
2005
).
The relation of patient satisfaction with complaints against physicians and malpractice lawsuits
.
The American Journal of Medicine
,
118
(
10
),
1126
33
.
Tamblyn
,
R.
,
Abrahamowicz
,
M.
,
Dauphinee
,
D.
,
Wenghofer
,
E.
,
Jacques
,
A.
,
Klass
,
D.
,
Smee
,
S.
,
Blackmore
,
D.
,
Winslade
,
N.
,
Girard
,
N.
,
Du Berger
,
R.
,
Bartman
,
I.
,
Buckeridge
,
D.L.
,
Hanley
,
J. A.
(
2007
).
Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities
.
JAMA: The Journal of the American Medical Association
,
298
(
9
),
993
1001
.
Tibble
,
H. M.
,
Broughton
,
N. S.
,
Studdert
,
D. M.
,
Spittal
,
M. J.
,
Hill
,
N.
,
Morris
,
J. M.
, &
Bismark
,
M. M.
(
2017
).
Why do surgeons receive more complaints than their physician peers?
ANZ Journal of Surgery
.
87
(
9
).
Gender: Discipline
Alam
,
A.
,
Khan
,
J.
,
Liu
,
J.
,
Klemensberg
,
J.
,
Griesman
,
J.
, &
Bell
,
C. M.
(
2013
).
Characteristics and rates of disciplinary findings amongst anesthesiologists by professional colleges in Canada
.
Canadian Journal of Anesthesia
,
60
(
10
),
1013
1019
.
Alam
,
A.
,
Klemensberg
,
J.
,
Griesman
,
J.
, &
Bell
,
C. M.
(
2011
).
The characteristics of physicians disciplined by professional colleges in Canada
.
Open Medicine: A Peer-Reviewed, Independent, Open-Access Journal
,
5
(
4
),
e166
72
.
Alam
,
A.
,
Kurdyak
,
P.
,
Klemensberg
,
J.
,
Griesman
,
J.
, &
Bell
,
C. M.
(
2012
).
The Characteristics of psychiatrists disciplined by professional colleges in Canada
.
PLoS ONE
,
7
(
11
),
e50558
.
Elkin
,
K.
,
Spittal
,
M.
, &
Studdert
,
D.
(
2012
).
Risks of complaints and adverse disciplinary findings against international medical graduates in Victoria and Western Australia
.
The Medical Journal of Australia
,
197
(
8
),
448
452
.
Gender: Prescribing and Ordering Tests
Beers
,
M.
,
Fingold
,
S.
,
Ouslander
,
J.
,
Reuben
,
D.
,
Morgenstern
H
, &
Beck
,
J.
(
1993
).
Characteristics and quality of prescribing by doctors practicing in nursing homes
.
Journal of the American Geriatrics Society
,
41
(
8
),
802
807
.
Chauvel
,
N.
,
Le Vaillant
,
M.
, &
Pelletier-Fleury
,
N.
(
2013
).
Variation in HbA1c prescription for patients with diabetes in French general practice: An observational study prior to the implementation of a P4P programme
.
European Journal of Public Health
,
23
(
1
),
61
66
.
Exam Scores
Asch
,
D. A.
,
Nicholson
,
S.
,
Srinivas
,
S.
,
Herrin
,
J.
, &
Epstein
,
A. J.
(
2009
).
Evaluating obstetrical residency programs using patient outcomes
.
JAMA: The Journal of the American Medical Association
,
302
(
12
),
1277
1283
.
Champlain
,
A. F. De
,
Streefkerk
,
C.
,
Tian
,
F.
,
Roy
,
M.
, &
Qin
,
S.
(
2013
).
Predicting Family Medicine Specialty Certification Status Using Standardized Measures for a Sample of International Medical Graduates Engaged in a Practice-ready Assessment Pathway to Provisional Licensure
,
53
(
9
),
1689
1699
.
Cuddy
,
M. M.
,
Young
,
A.
,
Gelman
,
A.
,
Swanson
,
D. B.
,
Johnson
,
D. A.
,
Dillon
,
G. F.
, &
Clauser
,
B. E.
(
n.d.
).
Exploring the Relationships Between USMLE Performance and Disciplinary Action in Practice: A Validity Study of Score Inferences From a Licensure Examination
,
Academic Medicine
(
PDF only
),
1
6
.
Norcini
,
J. J.
,
Boulet
,
J. R.
,
Opalek
,
A.
, &
Dauphinee
,
W. D.
(
2014
).
The relationship between licensing examination performance and the outcomes of care by international medical school graduates
.
Academic Medicine: Journal of the Association of American Medical Colleges
,
89
(
8
),
1157
62
.
Schabort
,
I.
,
Mercuri
,
M.
, &
Grierson
,
L. E. M.
(
2014
).
Predicting international medical graduate success on college certification examinations: Responding to the Thomson and Cohl judicial report on IMG selection
.
Canadian Family Physician
,
60
(
10
),
e478
e484
.
Tamblyn
,
R.
,
Abrahamowicz
,
M.
,
Brailovsky
,
C.
,
Grand'Maison
,
P.
,
Lescop
,
J.
,
Norcini
,
J.
,
Girard
,
N.
,
Haggerty
,
J.
(
1998
).
Association between licensing examination scores and resource use and quality of care in primary care practice
.
JAMA: The Journal of the American Medical Association
,
280
(
11
),
989
996
.
Tamblyn
,
R.
,
Abrahamowicz
,
M.
,
Dauphinee
,
W. D.
,
Hanley
,
J. A.
,
Norcini
,
J.
,
Girard
,
N.
,
Grand'Maison
,
P.
,
Brailovsky
,
C.
(
2002
).
Association between lincensure examination scores and practice in primary care
.
Journal of the American Medical Association
,
288
(
23
),
3019
3026
.
Tamblyn
,
R.
,
Abrahamowicz
,
M.
,
Dauphinee
,
D.
,
Wenghofer
,
E.
,
Jacques
,
A.
,
Klass
,
D.
,
Smee
,
S.
,
Blackmore
,
D.
,
Winslade
,
N.
,
Girard
,
N.
,
Du Berger
,
R.
,
Bartman
,
I.
,
Buckeridge
,
D.L.
,
Hanley
,
J. A.
(
2007
).
Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities
.
JAMA: The Journal of the American Medical Association
,
298
(
9
),
993
1001
.
Wenghofer
,
E.
,
Klass
,
D.
,
Abrahamowicz
,
M.
,
Dauphinee
,
D.
,
Jacques
,
A.
,
Smee
,
S.
,
Blackmore
,
D.
,
Winslade
,
N.
,
Reidel
,
K.
,
Bartman
,
I.
,
Tamblyn
,
R.
(
2009
).
Doctor scores on national qualifying examinations predict quality of care in future practice
.
Medical Education
,
43
(
12
),
1166
73
.
Specialty: Complaints
Bismark
,
M. M.
,
Spittal
,
M. J.
,
Gurrin
,
L. C.
,
Ward
,
M.
, &
Studdert
,
D. M.
(
2013
).
Identification of doctors at risk of recurrent complaints: a national study of healthcare complaints in Australia
.
BMJ Quality & Safety
,
22
(
7
),
532
540
.
Bismark
,
M. M.
,
Spittal
,
M. J.
, &
Studdert
,
D. M.
(
2011
).
Prevalence and characteristics of complaint-prone doctors in private practice in Victoria
.
The Medical Journal of Australia
,
195
(
1
),
25
8
.
Elkin
,
K.
,
Spittal
,
M.
, &
Studdert
,
D.
(
2012
).
Risks of complaints and adverse disciplinary findings against international medical graduates in Victoria and Western Australia
.
Medical Journal
,
197
,
448
452
.
Enbom
,
J. A.
, &
Thomas
,
C. D.
(
1997
).
Evaluation of sexual misconduct complaints: The Oregon Board of Medical Examiners, 1991 to 1995
.
American Journal of Obstetrics and Gynecology
,
176
(
6
),
1340
1348
.
Hickson
,
G. B.
,
Federspiel
,
C. F.
,
Blackford
,
J.
,
Pichert
,
J. W.
,
Gaska
,
W.
,
Merrigan
,
M. W.
, &
Miller
,
C. S.
(
2007
).
Patient Complaints and Malpractice Risk in a Regional Healthcare Center
.
Southern Medical Journal
,
100
(
8
),
791
796
.
Hickson
,
G. B.
,
Federspiel
,
C. F.
,
Pichert
,
J. W.
,
Miller
,
C. S.
,
Gauld-Jaeger
,
J.
, &
Bost
,
P.
(
2002
).
Patient complaints and malpractice risk
.
JAMA: The Journal of the American Medical Association
,
287
(
22
),
2951
2957
.
Spittal
,
M. J.
,
Bismark
,
M. M.
, &
Studdert
,
D. M.
(
2015
).
The PRONE score: an algorithm for predicting doctors' risks of formal patient complaints using routinely collected administrative data
.
BMJ Quality & Safety
,
24
(
6
),
360
8
.
Tamblyn
,
R.
,
Abrahamowicz
,
M.
,
Dauphinee
,
D.
,
Wenghofer
,
E.
,
Jacques
,
A.
,
Klass
,
D.
,
Smee
,
S.
,
Blackmore
,
D.
,
Winslade
,
N.
,
Girard
,
N.
,
Du Berger
,
R.
,
Bartman
,
I.
,
Buckeridge
,
D.L.
,
Hanley
,
J. A.
(
2007
).
Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities
.
JAMA: The Journal of the American Medical Association
,
298
(
9
),
993
1001
.
Tibble
,
H. M.
,
Broughton
,
N. S.
,
Studdert
,
D. M.
,
Spittal
,
M. J.
,
Hill
,
N.
,
Morris
,
J. M.
, &
Bismark
,
M. M.
(
2017
).
Why do surgeons receive more complaints than their physician peers?
ANZ Journal of Surgery
.
87
(
9
).
Specialty: Discipline
Alam
,
A.
,
Khan
,
J.
,
Liu
,
J.
,
Klemensberg
,
J.
,
Griesman
,
J.
, &
Bell
,
C. M.
(
2013
).
Characteristics and rates of disciplinary findings amongst anesthesiologists by professional colleges in Canada
.
Canadian Journal of Anesthesia
,
60
(
10
),
1013
1019
.
Alam
,
A.
,
Klemensberg
,
J.
,
Griesman
,
J.
, &
Bell
,
C. M.
(
2011
).
The characteristics of physicians disciplined by professional colleges in Canada
.
Open Medicine: A Peer-Reviewed, Independent, Open-Access Journal
,
5
(
4
),
e166
72
.
Alam
,
A.
,
Kurdyak
,
P.
,
Klemensberg
,
J.
,
Griesman
,
J.
, &
Bell
,
C. M.
(
2012
).
The Characteristics of psychiatrists disciplined by professional colleges in Canada
.
PLoS ONE
,
7
(
11
),
e50558
.

PROBABLE RISK FACTORS

PROBABLE RISK FACTORS
Practice Experience: Discipline
Alam
,
A.
,
Khan
,
J.
,
Liu
,
J.
,
Klemensberg
,
J.
,
Griesman
,
J.
, &
Bell
,
C. M.
(
2013
).
Characteristics and rates of disciplinary findings amongst anesthesiologists by professional colleges in Canada
.
Canadian Journal of Anesthesia
,
60
(
10
),
1013
1019
.
Alam
,
A.
,
Klemensberg
,
J.
,
Griesman
,
J.
, &
Bell
,
C. M.
(
2011
).
The characteristics of physicians disciplined by professional colleges in Canada
.
Open Medicine: A Peer-Reviewed, Independent, Open-Access Journal
,
5
(
4
),
e166
72
.
Alam
,
A.
,
Kurdyak
,
P.
,
Klemensberg
,
J.
,
Griesman
,
J.
, &
Bell
,
C. M.
(
2012
).
The Characteristics of psychiatrists disciplined by professional colleges in Canada
.
PLoS ONE
,
7
(
11
),
e50558
.
Practice Experience: Other Performance Outcomes
Choudhry
,
N. K.
,
Fletcher
,
R. H.
, &
Soumerai
,
S. B.
(
2005
).
The relationship between clinical experience and quality of health care
.
Annals of Internal Medicine
,
142
(
4
),
260
273
.
Workload
Chauvel
,
N.
,
Le Vaillant
,
M.
, &
Pelletier-Fleury
,
N.
(
2013
).
Variation in HbA1c prescription for patients with diabetes in French general practice: An observational study prior to the implementation of a P4P programme
.
European Journal of Public Health
,
23
(
1
),
61
66
.
Dean
,
B.
,
Schachter
,
M.
,
Vincent
,
C.
, &
Barber
,
N.
(
2002
).
Causes of prescribing errors in hospital inpatients: a prospective study
.
Lancet
,
359
(
9315
),
1373
1378
.
Kline
,
T. J. B.
,
Willness
,
C.
, &
Ghali
,
W. A.
(
2008
).
Predicting patient complaints in hospital settings
.
Qual Saf Heatlth Care
,
17
,
346
350
.
Wenghofer
,
E. F.
,
Campbell
,
C.
,
Marlow
,
B.
,
Kam
,
S. M.
,
Carter
,
L.
, &
McCauley
,
W.
(
2015
).
The effect of continuing professional development on public complaints: a case-control study
.
Medical Education
,
49
(
3
),
264
75
.
Wenghofer
,
E.
,
Williams
,
A.
, &
Klass
,
D.
(
2009
).
Factors Affecting Physician Performance: Implications for Performance Improvement and Governance Facteurs qui influent sur le rendement des médecins: répercussions pour l ' amélioration du rendement et pour la gouvernance
.
Healthcare Policy
,
5
(
2
),
141
160
.
Solo Practice
Brekke
,
M.
,
Rognstad
,
S.
,
Straand
,
J.
,
Furu
,
K.
,
Gjelstad
,
S.
,
Bjørner
,
T.
, &
Dalen
,
I.
(
2008
).
Pharmacologically inappropriate prescriptions for elderly patients in general practice: How common? Baseline data from The Prescription Peer Academic Detailing (Rx-PAD) study
.
Scandinavian Journal of Primary Health Care
,
26
(
2
),
80
85
.
Chauvel
,
N.
,
Le Vaillant
,
M.
, &
Pelletier-Fleury
,
N.
(
2013
).
Variation in HbA1c prescription for patients with diabetes in French general practice: An observational study prior to the implementation of a P4P programme
.
European Journal of Public Health
,
23
(
1
),
61
66
.
Grace
,
E. S.
,
Wenghofer
,
E. F.
, &
Korinek
,
E. J.
(
2014
).
Predictors of Physician Performance on Competence Assessment
.
Academic Medicine
,
89
(
6
),
912
919
.
McAuley
,
R. G.
,
Paul
,
W. M.
,
Morrison
,
G. H.
,
Beckett
,
R. F.
, &
Goldsmith
,
C. H.
(
1990
).
Five-year results of the peer assessment program of the College of Physicians and Surgeons of Ontario
.
CMAJ
.
143
(
11
),
1193
9
Norcini
,
J. J.
,
Fletcher
,
S. W.
,
Quimby
,
B. B.
, &
Shea
,
J. A.
(
1985
).
Performance of women candidates on the American Board of Internal Medicine Certifying Examination, 1973–1982
.
Annals of Internal Medicine
,
102
(
1
),
115
118
.
Xierali
,
I. M.
,
Rinaldo
,
J.
,
Green
,
L.
,
Petterson
,
S.
,
Phillips
,
R. L.
,
Bazemore
,
A.
,
Newton
,
W. P.
,
Puffer
,
J. C.
(
2011
).
Family Physician Participation in Maintenance of Certification
.
Annals of Family Medicine
,
9
(
3
),
203
210
.
Lack Of Engagement/Patient Centeredness
Kain
,
N.
,
Hodwitz
,
K.
,
Yen
,
W.
&
Ashworth
,
N.
(
2018
).
Experiential Knowledge of Risk and Support Factors for Physician Performance in Canada: A Qualitative Study
.
BMJ Open
.
Volume 9
(
2
).

POSSIBLE RISK FACTORS

POSSIBLE RISK FACTORS
IMG: Educational Performance
Andrew
,
R. F.
(
2010
).
How do IMGs compare with Canadian medical school graduates in a family practice residency program?
Can Fam.Physician.
,
56
(
9
),
e318
e322
.
Benson
,
J. A.
,
Meskauskas
,
J. A.
, &
Grosso
,
L. J.
(
1981
).
Performance of U.S. citizen-foreign medical graduates on certifying examinations in internal medicine
.
The American Journal of Medicine
,
71
(
2
),
270
273
.
Garibaldi
,
R. A.
,
Subhiyah
,
R.
,
Moore
,
M. E.
, &
Waxman
,
H.
(
2002
).
The In-Training Examination in Internal Medicine: An analysis of resident performance over time
.
Annals of Internal Medicine
,
137
(
6
),
505
510
.
MacLellan
,
A. M.
,
Brailovsky
,
C.
,
Rainsberry
,
P.
,
Bowmer
,
I.
, &
Desrochers
,
M.
(
2010
).
Examination outcomes for international medical graduates pursuing or completing family medicine residency training in Quebec
.
Can Fam.Physician.
,
56
(
9
),
912
918
.
McManus
,
I. C.
, &
Wakeford
,
R.
(
2014
).
PLAB and UK graduates' performance on MRCP (UK) and MRCGP examinations: data linkage study
,
2621
(
April
),
1
24
.
Norcini
,
J. J.
,
Boulet
,
J. R.
,
Whelan
,
G. P.
, &
McKinley
,
D. W.
(
2005
).
Specialty board certification among U.S. citizen and non-U.S. citizen graduates of international medical schools
.
Academic Medicine: Journal of the Association of American Medical Colleges
,
80
(
10 Suppl
),
S42
5
.
IMG: Complaints
Bismark
,
M. M.
,
Spittal
,
M. J.
, &
Studdert
,
D. M.
(
2011
).
Prevalence and characteristics of complaint-prone doctors in private practice in Victoria
.
The Medical Journal of Australia
,
195
(
1
),
25
28
.
Elkin
,
K.
,
Spittal
,
M.
, &
Studdert
,
D.
(
2012
).
Risks of complaints and adverse disciplinary findings against international medical graduates in Victoria and Western Australia
.
The Medical Journal of Australia
,
197
(
8
),
448
452
.
Tamblyn
,
R.
,
Abrahamowicz
,
M.
,
Dauphinee
,
D.
,
Wenghofer
,
E.
,
Jacques
,
A.
,
Klass
,
D.
,
Smee
,
S.
,
Blackmore
,
D.
,
Winslade
,
N.
,
Girard
,
N.
,
Du Berger
,
R.
,
Bartman
,
I.
,
Buckeridge
,
D.L.
,
Hanley
,
J. A.
(
2007
).
Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities
.
JAMA: The Journal of the American Medical Association
,
298
(
9
),
993
1001
.
IMG: Discipline
Alam
,
A.
,
Matelski
,
J. J.
,
Goldberg
,
H. R.
,
Liu
,
J. J.
,
Klemensberg
,
J.
, &
Bell
,
C. M.
(
2017
).
The Characteristics of International Medical Graduates Who Have Been Disciplined by Professional Regulatory Colleges in Canada: A Retrospective Cohort Study
.
Academic Medicine: Journal of the Association of American Medical Colleges
,
92
(
2
),
244
249
.
Alam
,
A.
,
Klemensberg
,
J.
,
Griesman
,
J.
, &
Bell
,
C. M.
(
2011
).
The characteristics of physicians disciplined by professional colleges in Canada
.
Open Medicine: A Peer-Reviewed, Independent, Open-Access Journal
,
5
(
4
),
e166
72
.
Cardarelli
,
R.
, &
Licciardone
,
J. C.
(
2006
).
Factors Associated With High-Severity Disciplinary Action by a State Medical Board: A Texas Study of Medical License Revocation
.
The Journal of the American Osteopathic Association
,
106
(
3
),
153
156
.
Elkin
,
K.
,
Spittal
,
M.
, &
Studdert
,
D.
(
2012
).
Risks of complaints and adverse disciplinary findings against international medical graduates in Victoria and Western Australia
.
The Medical Journal of Australia
,
197
(
8
),
448
452
.
Humphrey
,
C.
,
Hickman
,
S.
, &
Gulliford
,
M. C.
(
2011
).
Place of medical qualification and outcomes of UK General Medical Council “fitness to practise” process: cohort study
.
BMJ
.,
342
,
d1817
.
Kohatsu
,
N. D.
,
Gould
,
D.
,
Ross
,
L. K.
, &
Fox
,
P. J.
(
2004
).
Characteristics associated with physician discipline: a case-control study
.
Arch.Intern.Med.
,
164
(
6
),
653
658
.
IMG: Other Performance Outcomes
Ko
,
D. T.
,
Austin
,
P. C.
,
Chan
,
B. T. B.
, &
Tu
,
J. V.
(
2005
).
Quality of care of international and Canadian medical graduates in acute myocardial infarction
.
Archives of Internal Medicine
,
165
(
4
),
458
63
.
Norcini
,
J. J.
,
Boulet
,
J. R.
,
Dauphinee
,
W. D.
,
Opalek
,
A.
,
Krantz
,
I. D.
, &
Anderson
,
S. T.
(
2010
).
Evaluating the quality of care provided by graduates of international medical schools
.
Health Affairs
.
29
(
8
),
1461
1468
.
Tsugawa
,
Y.
,
Jena
,
A. B.
,
Orav
,
E. J.
, &
Jha
,
A. K.
(
2017
).
Quality of care delivered by general internists in US hospitals who graduated from foreign versus US medical schools: observational study
.
BMJ
,
356
,
1
8
.
Zaheer
,
S.
,
Pimentel
,
S. D.
,
Simmons
,
K. D.
,
Kuo
,
L. E.
,
Datta
,
J.
,
Williams
,
N.
,
Fraker
,
D.L.
,
Kelz
,
R. R.
(
2017
).
Comparing International and United States Undergraduate Medical Education and Surgical Outcomes Using a Refined Balance Matching Methodology
.
Annals of Surgery
,
265
(
5
),
916
922
.

About the Authors

Wendy Yen, PhD(c), is a Senior Researcher at the College of Physicians and Surgeons of Ontario (CPSO).

Niels Thakkar, MSc, is Research Associate at the CPSO.