ABSTRACT
Despite rapid growth in the number of physicians and academic institutions entering the field of global health, there are few tools that inform global health curricula and assess physician readiness for this field.
To address this gap, we describe the development and pilot testing of a new tool to assess nontechnical competencies and values in global health. Competencies assessed include systems-based practice, interpersonal and cross-cultural communication, professionalism and self-care, patient care, mentoring, teaching, management, and personal motivation and experience.
The Global Health Delivery Competency Assessment Tool presents 15 case vignettes and open-ended questions related to situations a global health practitioner might encounter, and grades the quality of responses on a 6-point ordinal scale. We interviewed 17 of 18 possible global health residents (94%), matched with 17 residents not training in global health, for a total of 34 interviews. A second reviewer independently scored recordings of 13 interviews for reliability.
Pilot testing indicated a high degree of discriminant validity, as measured by the instrument's ability to distinguish between residents who were and were not enrolled in a global health program (P < .001). It also demonstrated acceptable consistency, as assessed by interrater reliability (κ = 0.53), with a range of item-level agreement from 84%–96%.
The tool has potential applicability to a variety of academic and programmatic activities, including evaluation of candidates for global health positions and evaluating the success of training programs in equipping practitioners for entry into this field.
Despite growing interest in global health, there is a lack of relevant curricula and assessment of physician readiness for the field.
A new tool assesses competencies and values in global health, including systems-based practice and interpersonal and cross-cultural communications.
Single institution, single specialty study with a modest sample size, and a lack of true outcome measures may limit generalizability.
The tool is relevant to programmatic activities, including evaluation of candidates and assessing the success of training programs in equipping practitioners for practice in this field.
Introduction
An increasing number of US medical trainees and faculty are working in low-income countries, with many faculty integrating this work into their academic careers.1–3 Despite increased involvement in global health, the capacity of many academic institutions to support global health faculty, residents, and students has lagged behind.1,4
The field of global health, as the multidisciplinary study of clinical care and the systems required to deliver health care in resource-poor settings, has limited consensus on how to structure curricula or practical experiences.1,4 For example, schools of public health show considerable variability in required competencies, despite the existence of examples, including 1 from the Association of Schools and Programs of Public Health (ASPPH).5,6 A recent systematic review concluded that there was a marked absence of curricula across medical schools for teaching global health to future physicians.7
Domains identified by the ASPPH contain several nontechnical competencies and values (competencies of critical importance that are not related to science or technology).6,8 The review7 found that global health curricula strongly emphasized technical competencies, and other literature also suggests that current curricula place less emphasis on nontechnical competencies and values.6,9,10
When deployed to the field, global health practitioners work in settings with limited resources and severe disease burden, facing challenges for which they often are not prepared. Nontechnical challenges arise from different socioeconomic and cultural norms (especially around topics like gender and sexuality) or assignments beyond clinical work, such as program management, for which global health trainees have not been formally prepared.11–14 A few institutions have acknowledged this shortcoming, incorporating either nontechnical competencies or cultural competency into their global health curricula.11,15,16
We do not know whether these curricula produce physicians prepared to thrive as global health practitioners. Furthermore, nontechnical competencies and values are more challenging to assess than technical competencies.4
The purpose of this article is to describe the development and pilot testing of a tool for assessing these nontechnical competencies and values.
Methods
Development of the Global Health Delivery Competency Assessment Tool
In 2004, Brigham and Women's Hospital (BWH) launched a global health equity residency, a 4-year program with a curriculum in global health and internal medicine that trains physicians as clinicians, managers, and implementers to address health disparities.17 We developed a scenario-based tool to assess nontechnical competencies and values for physicians planning a career in global health.
We chose a scenario-based approach because it has several advantages over other approaches. First, it relates to the actual experience of global health by presenting real-life situations.18 Second, it removes common test-taking strategies for arriving at a correct answer, such as the process of elimination.19 Third, interviewees are challenged to provide a response, based on both the experience with which they are confronted and their own experience for navigating complex social situations.20 This lens is particularly appropriate given the cross-cultural and context-specific experiences encountered in global health. Other methods (direct observation in the field or simulation) were theoretically possible but were considered less logistically feasible.
The Global Health Delivery Competency Assessment Tool (GHD-CAT) was designed to evaluate preparedness for settings with limited resources, severe disease burden, and different socioeconomic and cultural norms. The GHD-CAT was developed in 3 stages: a literature review, an examination of existing tools, and interviews with key informants.
In the first phase, we searched PubMed (National Center for Biotechnology Information, Bethesda, MD) for publications on global health curricula and assessment tools. Sensitivity of searches was improved through the use of key words and bibliographies of eligible studies, as well as through an extended review of documents produced by organizations outside traditional academic channels.
The search identified 5 relevant frameworks: the Ontario Global Health Family Medicine Curriculum Working Group,11 the Global Health Education Consortium and the Association of Faculties of Medicine of Canada's Global Health Resource Group Core Competencies Project,21 the Global Health Competency Model of the Association of Schools of Public Health,6 the University of Minnesota Family Medicine Residency Global Health Curriculum,15 and the University of Washington International Health Program Competencies.9 The search did not identify any formal assessment tools for nontechnical competencies and values in the peer-reviewed literature.
The authors reviewed the frameworks and identified recurrent themes, and consolidated the content into competencies. Case vignettes were collaboratively developed, comprising an initial version of the GHD-CAT.
Key informant interviews with 8 global health experts addressed their experiences working with trainees in global health. The key informants described scenarios related to communication skills, professionalism, and working in cross-cultural settings. Feedback was recorded. The 3 phases were conducted in an iterative process, with further revision to the GHD-CAT. In the final stage, key informants were given the opportunity to review the GHD-CAT and provide feedback.
The interviews had 15 questions with 6 possible points each, for a total possible score of 90. Competencies assessed by the GHD-CAT were aligned with Accreditation Council for Graduate Medical Education competencies, similar to previous approaches,15 including systems-based practice, interpersonal and cross-cultural communication, professionalism and self-care, clinical care, mentoring, teaching, management, and personal motivation and experience (table). Each case vignette assessed a specific competency and had a corresponding scoring rubric for excellent (5–6 points), good (3–4 points), or limited (1–2 points), provided as online supplemental material.
Pilot Testing the GHD-CAT
We pilot tested the GHD-CAT with 2 cohorts: global health and non–global health residents at BWH. Participants volunteered by responding to an e-mail invitation sent to all internal medicine residents. Participants were assigned random alphanumeric codes, and no identifying information was recorded. All participants provided oral informed consent.
The same individual conducted all interviews, which were performed in person, in a private setting, and without observers. Thirty-four interviews were conducted, 50% with global health education (GHE) physician residents and 50% with non-GHE physician residents. A total of 17 of the 18 current GHE resident physicians (94%) participated. To measure the GHD-CAT reliability, 13 interviews were randomly selected to be audio recorded and were scored by an individual who was blinded to participants' background.
Ethical approval was received by the Institutional Review Board at Partners Healthcare.
Average scores for each competency and total scores were reported according to the following classifications: (1) GHE residents; (2) non-GHE residents; (3) GHE residents in years 1–2; and (4) GHE residents in years 3–4. Median scores across all 4 classifications were compared using 2-sided Wilcoxon rank sum tests for ordinal values, with a cutoff of α = .05.
Interrater reliability was assessed with scores from the initial and second interviewer using a quadratic-weighted Cohen kappa coefficient (κ) to measure the degree of concordance.
Results
Demographic characteristics of the interview sample showed the 2 cohorts of residents were similar in terms of age, sex, and residency year.
The average total score on the GHD-CAT for GHE residents was 79.2, versus 58.6 for non-GHE residents (z = 4.16; 95% confidence interval [CI] 1.87–5.20; P < .001). This difference remained after controlling for sex and age of respondents (z = 3.82; 95% CI 1.63–5.05; P < .001). The GHE residents scored significantly higher than non-GHE residents did on all thematic areas, ranging from 29% for mentoring, teaching, and management, to 62% for motivation and experience (P = .008; figure). The GHE residents in years 1 or 2 of the program had an average score of 75.2, and GHE residents in years 3 or 4 had an average score of 83.6 (z = 2.31; 95% CI 0.35–4.26; P = .028).
Average Item Score by Resident Type by the Global Health Delivery Competency Assessment Tool Category
Abbreviations: IM, internal medicine; GHE, global health education.
Average Item Score by Resident Type by the Global Health Delivery Competency Assessment Tool Category
Abbreviations: IM, internal medicine; GHE, global health education.
Interrater reliability across 13 interviews was κ = 0.53, connoting an average overall level of agreement across all questions of 93.5% versus an expected level of agreement of 86.0% (z = 7.96, P < .001). Item-level agreement ranged from a low of 84% to a high of 96%. These results indicate an acceptable and statistically significant level of concordance.22
Participants provided feedback at the end of interviews, revealing that the case vignettes were realistic and familiar to those with global health experience. Participants reported that the conversation was enjoyable and informative.
Discussion
Quantitative and qualitative evidence from pilot testing of the GHD-CAT found evidence of validity, reliability, and acceptability. While our sample size was modest, the assessment tool performed well in discriminating between individuals with experience in global health and those without such experience. Among GHE residents, the tool showed significantly higher scores for senior trainees compared with junior trainees.
We anticipate that this assessment tool could have direct utility and application in the following areas:
- 1.
Curriculum and Training Evaluation: Use of the GHD-CAT to assess whether a program is effectively incorporating nontechnical competencies and values.
- 2.
Job Candidate Evaluation: Use of the tool by nongovernmental organizations, governments, and academic institutions to determine whether a candidate demonstrates preparedness for practice in global health, beyond technical competencies.
- 3.
Practical Training: Use of the GHD-CAT as a scenario-based learning tool to help expose trainees to real-world case vignettes and to provide guidance on where ongoing support is needed.
In the context of training GHE residents, the tool could be used to examine improvements in training programs over time, by studying whether cohorts' scores increase from 1 year to the next. To date, there is an absence of assessment tools for evaluating the effectiveness of global health curricula that teach nontechnical competencies and values relevant to global health practitioners around the world.7,23 With further evaluation, the GHD-CAT may help bridge this divide.
Participants with experience in global health reported that the case vignettes reflected comparable scenarios to the ones experienced within the context of their work. For example, 1 interviewee remarked, “These scenarios are remarkably realistic, and remind me of my own experiences.”
Our evaluation of the GHD-CAT has some limitations. The sample size was modest (N = 34), and data were collected at a single institution and only from internal medicine residents, limiting generalizability. The tool has yet to be evaluated based on outcome measures, such as whether high scores will predict successful work in global health. Lastly, only 13 interviews were scored by a second reviewer, limiting our ability to measure interrater reliability. Future studies could incorporate repeat interviews to see whether results are consistent, explore different vignettes to see if similar results will be obtained, and investigate the ideal number of vignettes.24,25
Conclusion
The GHD-CAT is an innovative tool designed to assess the effectiveness of training in global health for individuals who intend to pursue a career in delivering health care in resource-poor global settings. Pilot testing of the GHD-CAT demonstrated evidence of validity and interrater reliability.
