This report has been made available for readers of the Journal of Medical Regulation courtesy of the Pan-Canadian Physician Factors Collaboration. Copyright and all rights reserved by the Pan-Canadian Physician Factors Collaboration.
Background
The Pan-Canadian Physician Factors Collaboration is a research and development consortium that seeks to identify the specific risk and support factors that affect physician performance across a variety of contexts. A risk factor is defined as a factor that negatively impacts any facet of performance, while a support factor helps to foster, develop or improve performance (Glover Takahashi, Nayer, & St Amant, 2017). Risk factors for poor performance have been studied in the published literature, generating evidence about the individual, organizational and systems-level factors that may hinder performance. Studies have also examined support factors that may promote high-quality practices. Since the beginning of the collaboration in 2015, there has been a concerted effort to develop a strong evidence base to support the evolution of regulatory assessment programs and safe medical practices.
Risk-based prinicples have recently become a central tenet of regulation (OECD, 2012). Risk-based approaches are being applied to proactive medical regulation in several Canadian jurisdictions and a strong evidence base is needed to support this work. For example, there is interest among Canadian regulators to assess more physicians, more frequently to ensure that physicians engage in lifelong learning and maintenance of competence throughout their careers. Given limited resources and the imperative to develop assessments that foster learning, there is a growing need to ensure that physicians are routed to tailored interactions with the regulator proportionate to risk and learning needs. One strategy to accomplish this is to route physicians to appropriate assessments based on a profile that summarizes the combination of risk and support factors in a physician's practice. This approach is in line with the tenets of Right Touch Regulation, which has been the cornerstone of medical regulation in the United Kingdom over the past few years (Professionals Standards Authority, 2015).
The Pan-Canadian Physician Factors Collaboration was started by Medical Regulatory Authorities (MRAs) in Canada but other physician organizations have since joined this work. It is recognized that numerous medical stakeholders have a vested interest in how higher-risk physicians are assessed and how physicians are supported in the system as they mitigate and manage risk. Knowledge of factors that affect physician performance is relevant and important for practicing physicians, patients, medical regulatory authorities, national certifying bodies and other stakeholder groups involved in the design and delivery of healthcare. As such, a collaboration with representatives from relevant physician organizations has been convened to guide this work. This group is charged with ensuring that the program is sustainable and that the knowledge produced helps to modernize performance assessments and continuing competence systems.
Over the past few years, the collaboration has invested in developing an evidence base on risk and support factors to physician performance to achieve the following across Canada:
Build physician self-awareness and learning about risk and support factors
Create protective practice strategies
Select high-risk physicians for regulatory assessments
Promote risk-based regulatory and medico-legal practices
Inform physician assessment and continuing professional development programs
As of 2019, the Pan-Canadian Physician Factors Collaboration has achieved stable program status under the leadership of the Medical Council of Canada (MCC) and the Federation of Medical Regulatory Authorities of Canada (FMRAC). This formal infrastructure will facilitate on-going collaboration, resource management and knowledge translation among system partners.
Purpose
This report is a state of the science summary of what is currently known about select risk and support factors to physician performance. The purpose of this work is to compile evidence from multiple reviews into one accessible and usable document.
This is an important endeavor as the literature base is vast and it is often difficult to reach conclusions about the extent to which the evidence supports the relationship between factors and practice performance.
The current paper strives to:
Summarize what is known about select risk and support factors to physician performance
Summarize what remains unknown
Provide recommendations for future research
Data Sources
The data sources included in this document and the associated methodologies are described below. Systematic reviews aim to comprehensively search for all relevant articles in an area of study. A quality screen is developed to determine which articles should be included or excluded based on explicit criteria and a standardized tool is used by multiple coders to evaluate articles (Grant & Booth, 2009). Systematic reviews are the gold standard for literature reviews but they are not always feasible due to resource constraints. It is important to note that the methodologies for each of the data sources described in this synthesis do not meet the requirements of a full systematic review although systematic review methods were used to critically appraise studies.
A scoping review was commissioned to ascertain the extent to which risk and support factors were being studied in the published literature (Glover Takahashi, et al., 2017). Scoping reviews are conducted to broadly identify key concepts, theories and sources of evidence, but do not synthesize findings from individual studies (Arksey & O'Malley, 2005). Moving beyond the scoping review to evidence synthesis, the McMaster Health Forum was commissioned to conduct a rapid synthesis of factors associated with competence. They examined 17 primary studies looking at risk and support factors for quality clinical practices across three performance outcomes: complaints, disciplinary action and prescribing and testing errors (Wilson & JN, 2015). Building off of the McMaster work, a more comprehensive systematized review of factors related to complaints was conducted because the rapid review did not capture all relevant articles due to time constraints. Systematized reviews were also conducted for the following factors: being an internationally trained medical graduate, age, and exam scores (note: these were not systematic reviews).
See Table 1 for an overview of evidence syntheses developed or commissioned by MRAs and definitions for different types of review methodologies.
In addition to the above, two additional data sources contributed valuable knowledge to this report. MRA research staff led a qualitative study examining risk and supports to physician performance based on the experiential knowledge of 23 physician assessors (Kain, et al., 2018). Additionally, a systematic review exploring the relationship between clinical experience and quality of health care was uncovered as part of this work (this review was not commissioned or developed by MRAs) (Choudhry, Fletcher, & Soumerai, 2005).
All of these activities represent the current “state of the science” as it pertains to consolidated knowledge of risk and support factors to physician performance. In total, over 900 published articles were coded by MRA researchers and commissioned associates to summarize what is known about risk and support factors and reveal gaps in the evidence base.
Methods
The purpose of this report is to consolidate evidence on risk and support factors to physician performance. This is an ambitious task due to the large body of evidence that exists in this area. The individual reviews included in this report were conducted sequentially between 2015–2018. The order of the reviews was designed purposefully so that earlier reviews could inform subsequent reviews in an iterative way. For example, the scoping review served as a seminal document to understand the extent to which factors were being studied in the published literature. From this, a rapid synthesis provided early insight into the key factors that are related to different performance outcomes. Rapid syntheses are limited by time constraints so additional systematized reviews were conducted for frequently studied factors in order to consolidate the evidence base. Articles uncovered in the scoping review served as starting point for each of the systematized reviews conducted by MRA researchers and commissioned associates.
For the state of the science summary, no additional searches were conducted as the data sources listed above served as inputs to this document. To consolidate the evidence from all data sources, two researchers (WY & NT) tabulated studies that found a relationship between a particular factor and a specific performance outcome (e.g. complaints, discipline, assessment). Combing across multiple syntheses and consolidating findings by factor allowed the research team to ascertain the degree to which the evidence supported a particular factor being associated with different performance outcomes. The research team focused on risk factors frequently studied in the published literature (age, gender, exam scores, specialty, workload, solo practice, international training and experience). Each risk factor was assigned a rating: factor, probable factor, and possible factor, based on the strength of the evidence supporting each factor. This was based on the quantity and consistency of the findings throughout the literature. The quality of article was not directly considered in the rating, as the researchers were consolidating syntheses which did not have uniform quality screens.
A system was created so that factors under investigation could be rated by the strength of the evidence (Table 2).
Results
Using the rating system described, age, gender, exam scores and specialty were coded as risk factors due to the strength of the evidence. Practice experience, workload and solo practice were coded as probable risk factors given that there were fewer studies on these factors but they were still trending in a consistent direction. IMG and lack of engagement were coded as a possible risk factor given that findings are conflicted for the former and only one study exists for the latter.
Table 3 provides a high-level summary of findings. Tables 4, 5 and 6 provide detailed findings as well as implications of the results. References for the detailed findings can be found in Appendix 2.
Discussion
To date, knowledge of risk and support factors that impact physician performance has never been consolidated in a comprehensive and usable way. It is envisioned that such knowledge will benefit physicians as well as physician organizations in the healthcare system. Physicians will be provided with knowledge to proactively mitigate risks in their practice before they arise. It is important to note that some risk factors are not modifiable and physicians should be encouraged to consider the information in the context of their own practices. MRAs will use knowledge of risk factors to triage physicians for assessment so that resources are used more efficiently to support risk-based regulation. National certifying colleges responsible for setting standards for training, certification and lifelong learning will use knowledge of factors to shape personal learning plans and develop programs that help physicians use supports to mitigate risk. Other stakeholders in the system, such as the Medical Council of Canada and the Canadian Medical Protective Association, will also use knowledge of factors to understand, support and protect physician practices. All of the above endeavors are geared towards promoting high-quality practices and an effective continuing competence system.
Several MRAs across Canada are already using consolidated knowledge of factors to inform regulatory processes and programs. British Columbia, Alberta, Ontario, Quebec and Nova Scotia are using knowledge of factors to select physicians for assessment. Quebec has been risk-prioritizing physicians for many years by using their in-house data to understand the associations between various demographic and practice factors and assessment outcomes. Other jurisdictions are begining to use the knowledge gained through Quebec's experience along with the evidence base to inform selection processes. Nova Scotia, Ontario and Alberta are also using knowledge of factors to develop assessment programming to inform physicians of risks and supports to practice. For example, Ontario is developing a self-directed online interactive tool that will provide physicians with an individualized practice profile of risk and support factors. A report will be made available through this process, which will provide physicians with mitigating strategies and resources available for practice improvement.
It should be highlighted that although this report consolidates evidence across markers of performance (e.g. complaints, discipline, assessment, etc.), it is always important to consider “at risk of what?”. Physician performance is a multi-dimensional construct and the quality of care delivered in one domain does not presuppose quality in another. Characteristics that are associated with physicians who have been disciplined may be different than those who have an unsuccessful assessment outcome. Both disciplinary action and assessment results are a measure of different aspects of poor performance. When applying the evidence in this summary, the context, purpose and what facet of care the program is measuring, assessing or facilitating needs to be considered.
Strengths
One of the key strengths of this work is that a comprehensive state of the science summary has never been conducted in this area to take stock of what is known. This work is valuable as evidence from multiple reviews is consolidated in a usable way to guide policy and program development. In the absence of such work, it is difficult to ascertain the strength of the evidence across multiple data sources. This work also represents a significant contribution in that considerable resources were dedicated to the individual inputs included in this report (i.e. numerous researchers and academic partners contributed to each of the individual syntheses). Through the scoping review, a comprehensive reference list of 754 studies that examined risk and support factors to physician competence was generated and will be used as a starting point to synthesize evidence about factors not covered in this report.
While the reviews included in the report do not meet the criteria for systematic review (with the exception of one), systematized methodology was used for each of the inputs to ensure rigor. Our understanding of select factors was enhanced by the ability to study their impact across a variety of performance outcomes (e.g. prescribing, assessment, complaints) as well as across various samples and settings due to the heterogenous nature of individual studies. In total, over 900 studies were coded as part of this analysis and we were able to ascertain the strength of the evidence for some well-studied risk factors as well as reveal gaps in the literature base. This consolidated work can now be shared with national and international audiences to inform risk-based approaches to regulation and continuing competence systems.
Limitations
Limitations also exist for the work described in this report. The methodology does not meet the rigor of a full systematic review, which means that some articles may have been missed, quality screens may or may not have been applied, depending on the review, and multiple coders were not always used. While this work represents a significant contribution, the research team was only able to synthesize evidence on select risk factors due to resource constraints. As such, this was not an exploratory exercise to catalogue all the possible factors, but rather, a confirmatory exercise to consolidate and summarize what is known. The impact of risk factors was considered for the performance outcomes most frequently studied in the published literature and does not represent all possible performance outcomes. As such, more work is needed to understand risk factors above and beyond what has been included in this report. For example, while the scoping review uncovered published articles on support factors, we have not taken an in-depth look at these studies and know relatively little compared to our knowledge of risk factors.
While the evidence base on risk and support factors to performance is growing steadily, there remains a scarcity of information on why identified factors may hinder or support practice. This dearth can be largely explained by the methodological approaches used to study risk and support factors to date; most of the published literature utilizes epidemiological approaches on epidemiological data, which does not shed light on why a factor might be a factor and typically only reveal correlations, not causal relationships. Finally, it is unorthodox to conduct a state of the science summary that includes syntheses that are not yet available in the public domain. We strategically decided to prioritize the current high-level synthesized findings of multiple reviews to facilitate sharing the knowledge with administrators and program developers in a timely manner.
Future Research Agenda
The evidence base will continue to grow as more factors research is published. A national research agenda will be developed and formalized under the direction of the Steering Committee and MCC/FMRAC to address gaps inclusive of the following:
Due to methodological constraints and availability of data, some factors are easier to study than others, which may contribute to why they are strong predictors of performance in the published literature (e.g. gender). There are likely risk and support factors that are strongly predictive of practice performance but are more difficult to measure and study (e.g. lack of engagement, patient centeredness). These important factors will need to be studied using different methodological approaches, such as qualitative research, which does not rely on administrative data.
Other factors not summarized in this report but identified in the literature include: lack of board or specialty certification, practice location, previous complaints or medico-legal events, practicing in a hospital environment and engaging in certain types of continuing professional development. Future work should examine evidence for these factors as well as study how organizational and system levels factors affect performance.
Qualitative methodologies will also help us to understand why a factor is a factor and not just whether or not it is associated with practice performance.
For factors where there is a lot of evidence (e.g. age, gender), future studies can stratify groups to garner more nuanced understanding. For example, stratifying by gender will allow researchers to understand the specific risk factors associated with being male. Stratifying by age cohorts will contribute to understanding factors pertinent to early, middle, and late career practice.
MRAs are well positioned to make a significant contribution to the existing evidence base due to the wealth of performance data that exists in these organizations.
Factors associated with performance outcomes not included in this review will need to be studied in future work (e.g. billing data, patient outcomes).
Compared to knowledge of risk factors, we know relatively little about factors that support high-quality practices. This will be a key area for future work as system partners focus on preventive strategies to facilitate the development of safe practices.
New directions for future research should focus on trends in health care inclusive of the role of context in performance, how to assess team-based care and the role of support factors in promoting high-quality practices.
Key Takeaways
To date, evidence of risk and supports to performance has never been consolidated in such a comprehensive way. The following represents key takeaways from this work:
The evidence base on risk and supports to physician practice will continuously evolve as practice changes and new knowledge is added to the system. It is the intention of the national collaboration to continue to update the evidence base so it can be iteratively applied to regulatory programming to ensure relevance.
It is always important to consider “at risk of what?” as risk factors differ across performance outcomes.
Audiences will need to bear in mind that the methodologies used in individual studies within each review examine associations between variables. This is not the same as examining the relationship between cause and effect among variables. As such, those with high-risk profiles will not necessarily perform more poorly, although they are more likely to.
Future national work will focus on developing the necessary infrastructure to share factors knowledge across MRAs, physician organizations and academic partners. The appropriate knowledge translation infrastructure is to be built under the guidance of the collaboration and MCC/FMRAC.
The knowledge generated in this document will need to be applied to processes and programs in the system to measure the added benefit to this approach (e.g., are triage tools developed to select at-risk physicians effective? Does providing physicians with knowledge of risk and support factors help to protect practices?).
Acknowledgments
This report was created on behalf of the Pan-Canadian Physician Factors Collaboration under the direction of the Co-chairs, Daniel Faulkner of the College of Physicians and Surgeons of Ontario and Dr. Karen Mazurek of the College of Physicians and Surgeons of Alberta. We thank Dr. Nigel Ashworth of the College of Physicians and Surgeons of Alberta for his contribution to the risk rating framework.
The following organizations are involved in the Canadian Physician Factors Collaboration:
College of Physicians and Surgeons of Alberta (CPSA)
College of Physicians and Surgeons of British Columbia (CPSBC)
College of Physicians and Surgeons of Manitoba (CPSM)
College of Physicians and Surgeons of Nova Scotia (CPSNS)
College of Physicians and Surgeons of Ontario (CPSO)
College des Medecins du Quebec (CMQ)
College of Family Physicians Canada (CFPC)
Canadian Medical Protective Association (CMPA)
Federation of Medical Regulatory Authorities of Canada (FMRAC)
Medical Council of Canada (MCC)
Royal College of Physicians and Surgeons of Canada (RCPSC)
General References
Appendix 1 Detailed Information of Data Sources
Appendix 2 References for Detailed Findings
- RISK FACTORS
16
- Age
16
- Complaints
16
- Physical Performance
16
- Cognitive Performance
16
- Psychological Wellness
16
- Clinical Knowledge or Performance
17
- Patient Safety
17
- Medico-Legal
18
- Gender
18
- Complaints
18
- Discipline
19
- Prescribing and Ordering Tests
19
- Exam Scores
19
- Specialty
19
- Complaints
19
- Discipline
20
- PROBABLE RISK FACTORS
20
- Practice Experience
20
- Discipline
20
- Other Performance Outcomes
20
- Workload
20
- Solo Practice
20
- Lack of Engagement/Patient Centeredness
20
- POSSIBLE RISK FACTORS
20
- IMG
20
- Educational Performance
20
- Complaints
21
- Discipline
21
- Other Performance Outcomes
21
