Background 

Despite rapid growth in the number of physicians and academic institutions entering the field of global health, there are few tools that inform global health curricula and assess physician readiness for this field.

Objective 

To address this gap, we describe the development and pilot testing of a new tool to assess nontechnical competencies and values in global health. Competencies assessed include systems-based practice, interpersonal and cross-cultural communication, professionalism and self-care, patient care, mentoring, teaching, management, and personal motivation and experience.

Methods 

The Global Health Delivery Competency Assessment Tool presents 15 case vignettes and open-ended questions related to situations a global health practitioner might encounter, and grades the quality of responses on a 6-point ordinal scale. We interviewed 17 of 18 possible global health residents (94%), matched with 17 residents not training in global health, for a total of 34 interviews. A second reviewer independently scored recordings of 13 interviews for reliability.

Results 

Pilot testing indicated a high degree of discriminant validity, as measured by the instrument's ability to distinguish between residents who were and were not enrolled in a global health program (P < .001). It also demonstrated acceptable consistency, as assessed by interrater reliability (κ = 0.53), with a range of item-level agreement from 84%–96%.

Conclusions 

The tool has potential applicability to a variety of academic and programmatic activities, including evaluation of candidates for global health positions and evaluating the success of training programs in equipping practitioners for entry into this field.

What was known and gap

Despite growing interest in global health, there is a lack of relevant curricula and assessment of physician readiness for the field.

What is new

A new tool assesses competencies and values in global health, including systems-based practice and interpersonal and cross-cultural communications.

Limitations

Single institution, single specialty study with a modest sample size, and a lack of true outcome measures may limit generalizability.

Bottom line

The tool is relevant to programmatic activities, including evaluation of candidates and assessing the success of training programs in equipping practitioners for practice in this field.

An increasing number of US medical trainees and faculty are working in low-income countries, with many faculty integrating this work into their academic careers.13  Despite increased involvement in global health, the capacity of many academic institutions to support global health faculty, residents, and students has lagged behind.1,4 

The field of global health, as the multidisciplinary study of clinical care and the systems required to deliver health care in resource-poor settings, has limited consensus on how to structure curricula or practical experiences.1,4  For example, schools of public health show considerable variability in required competencies, despite the existence of examples, including 1 from the Association of Schools and Programs of Public Health (ASPPH).5,6  A recent systematic review concluded that there was a marked absence of curricula across medical schools for teaching global health to future physicians.7 

Domains identified by the ASPPH contain several nontechnical competencies and values (competencies of critical importance that are not related to science or technology).6,8  The review7  found that global health curricula strongly emphasized technical competencies, and other literature also suggests that current curricula place less emphasis on nontechnical competencies and values.6,9,10 

When deployed to the field, global health practitioners work in settings with limited resources and severe disease burden, facing challenges for which they often are not prepared. Nontechnical challenges arise from different socioeconomic and cultural norms (especially around topics like gender and sexuality) or assignments beyond clinical work, such as program management, for which global health trainees have not been formally prepared.1114  A few institutions have acknowledged this shortcoming, incorporating either nontechnical competencies or cultural competency into their global health curricula.11,15,16 

We do not know whether these curricula produce physicians prepared to thrive as global health practitioners. Furthermore, nontechnical competencies and values are more challenging to assess than technical competencies.4 

The purpose of this article is to describe the development and pilot testing of a tool for assessing these nontechnical competencies and values.

Development of the Global Health Delivery Competency Assessment Tool

In 2004, Brigham and Women's Hospital (BWH) launched a global health equity residency, a 4-year program with a curriculum in global health and internal medicine that trains physicians as clinicians, managers, and implementers to address health disparities.17  We developed a scenario-based tool to assess nontechnical competencies and values for physicians planning a career in global health.

We chose a scenario-based approach because it has several advantages over other approaches. First, it relates to the actual experience of global health by presenting real-life situations.18  Second, it removes common test-taking strategies for arriving at a correct answer, such as the process of elimination.19  Third, interviewees are challenged to provide a response, based on both the experience with which they are confronted and their own experience for navigating complex social situations.20  This lens is particularly appropriate given the cross-cultural and context-specific experiences encountered in global health. Other methods (direct observation in the field or simulation) were theoretically possible but were considered less logistically feasible.

The Global Health Delivery Competency Assessment Tool (GHD-CAT) was designed to evaluate preparedness for settings with limited resources, severe disease burden, and different socioeconomic and cultural norms. The GHD-CAT was developed in 3 stages: a literature review, an examination of existing tools, and interviews with key informants.

In the first phase, we searched PubMed (National Center for Biotechnology Information, Bethesda, MD) for publications on global health curricula and assessment tools. Sensitivity of searches was improved through the use of key words and bibliographies of eligible studies, as well as through an extended review of documents produced by organizations outside traditional academic channels.

The search identified 5 relevant frameworks: the Ontario Global Health Family Medicine Curriculum Working Group,11  the Global Health Education Consortium and the Association of Faculties of Medicine of Canada's Global Health Resource Group Core Competencies Project,21  the Global Health Competency Model of the Association of Schools of Public Health,6  the University of Minnesota Family Medicine Residency Global Health Curriculum,15  and the University of Washington International Health Program Competencies.9  The search did not identify any formal assessment tools for nontechnical competencies and values in the peer-reviewed literature.

The authors reviewed the frameworks and identified recurrent themes, and consolidated the content into competencies. Case vignettes were collaboratively developed, comprising an initial version of the GHD-CAT.

Key informant interviews with 8 global health experts addressed their experiences working with trainees in global health. The key informants described scenarios related to communication skills, professionalism, and working in cross-cultural settings. Feedback was recorded. The 3 phases were conducted in an iterative process, with further revision to the GHD-CAT. In the final stage, key informants were given the opportunity to review the GHD-CAT and provide feedback.

The interviews had 15 questions with 6 possible points each, for a total possible score of 90. Competencies assessed by the GHD-CAT were aligned with Accreditation Council for Graduate Medical Education competencies, similar to previous approaches,15  including systems-based practice, interpersonal and cross-cultural communication, professionalism and self-care, clinical care, mentoring, teaching, management, and personal motivation and experience (table). Each case vignette assessed a specific competency and had a corresponding scoring rubric for excellent (5–6 points), good (3–4 points), or limited (1–2 points), provided as online supplemental material.

Pilot Testing the GHD-CAT

We pilot tested the GHD-CAT with 2 cohorts: global health and non–global health residents at BWH. Participants volunteered by responding to an e-mail invitation sent to all internal medicine residents. Participants were assigned random alphanumeric codes, and no identifying information was recorded. All participants provided oral informed consent.

The same individual conducted all interviews, which were performed in person, in a private setting, and without observers. Thirty-four interviews were conducted, 50% with global health education (GHE) physician residents and 50% with non-GHE physician residents. A total of 17 of the 18 current GHE resident physicians (94%) participated. To measure the GHD-CAT reliability, 13 interviews were randomly selected to be audio recorded and were scored by an individual who was blinded to participants' background.

Ethical approval was received by the Institutional Review Board at Partners Healthcare.

Average scores for each competency and total scores were reported according to the following classifications: (1) GHE residents; (2) non-GHE residents; (3) GHE residents in years 1–2; and (4) GHE residents in years 3–4. Median scores across all 4 classifications were compared using 2-sided Wilcoxon rank sum tests for ordinal values, with a cutoff of α = .05.

Interrater reliability was assessed with scores from the initial and second interviewer using a quadratic-weighted Cohen kappa coefficient (κ) to measure the degree of concordance.

Demographic characteristics of the interview sample showed the 2 cohorts of residents were similar in terms of age, sex, and residency year.

The average total score on the GHD-CAT for GHE residents was 79.2, versus 58.6 for non-GHE residents (z = 4.16; 95% confidence interval [CI] 1.87–5.20; P < .001). This difference remained after controlling for sex and age of respondents (z = 3.82; 95% CI 1.63–5.05; P < .001). The GHE residents scored significantly higher than non-GHE residents did on all thematic areas, ranging from 29% for mentoring, teaching, and management, to 62% for motivation and experience (P = .008; figure). The GHE residents in years 1 or 2 of the program had an average score of 75.2, and GHE residents in years 3 or 4 had an average score of 83.6 (z = 2.31; 95% CI 0.35–4.26; P = .028).

Interrater reliability across 13 interviews was κ = 0.53, connoting an average overall level of agreement across all questions of 93.5% versus an expected level of agreement of 86.0% (z = 7.96, P < .001). Item-level agreement ranged from a low of 84% to a high of 96%. These results indicate an acceptable and statistically significant level of concordance.22 

Participants provided feedback at the end of interviews, revealing that the case vignettes were realistic and familiar to those with global health experience. Participants reported that the conversation was enjoyable and informative.

Quantitative and qualitative evidence from pilot testing of the GHD-CAT found evidence of validity, reliability, and acceptability. While our sample size was modest, the assessment tool performed well in discriminating between individuals with experience in global health and those without such experience. Among GHE residents, the tool showed significantly higher scores for senior trainees compared with junior trainees.

We anticipate that this assessment tool could have direct utility and application in the following areas:

  • 1. 

    Curriculum and Training Evaluation: Use of the GHD-CAT to assess whether a program is effectively incorporating nontechnical competencies and values.

  • 2. 

    Job Candidate Evaluation: Use of the tool by nongovernmental organizations, governments, and academic institutions to determine whether a candidate demonstrates preparedness for practice in global health, beyond technical competencies.

  • 3. 

    Practical Training: Use of the GHD-CAT as a scenario-based learning tool to help expose trainees to real-world case vignettes and to provide guidance on where ongoing support is needed.

In the context of training GHE residents, the tool could be used to examine improvements in training programs over time, by studying whether cohorts' scores increase from 1 year to the next. To date, there is an absence of assessment tools for evaluating the effectiveness of global health curricula that teach nontechnical competencies and values relevant to global health practitioners around the world.7,23  With further evaluation, the GHD-CAT may help bridge this divide.

Participants with experience in global health reported that the case vignettes reflected comparable scenarios to the ones experienced within the context of their work. For example, 1 interviewee remarked, “These scenarios are remarkably realistic, and remind me of my own experiences.”

Our evaluation of the GHD-CAT has some limitations. The sample size was modest (N = 34), and data were collected at a single institution and only from internal medicine residents, limiting generalizability. The tool has yet to be evaluated based on outcome measures, such as whether high scores will predict successful work in global health. Lastly, only 13 interviews were scored by a second reviewer, limiting our ability to measure interrater reliability. Future studies could incorporate repeat interviews to see whether results are consistent, explore different vignettes to see if similar results will be obtained, and investigate the ideal number of vignettes.24,25 

The GHD-CAT is an innovative tool designed to assess the effectiveness of training in global health for individuals who intend to pursue a career in delivering health care in resource-poor global settings. Pilot testing of the GHD-CAT demonstrated evidence of validity and interrater reliability.

1
Drain
PK,
Primack
A,
Hunt
DD,
et al.
Global health in medical education: a call for more training and opportunities
.
Acad Med
.
2007
;
82
(
3
):
226
230
.
2
Merritt
MG
Jr,
Railey
CJ,
Levin
SA,
et al.
Involvement abroad of US academic health centers and major teaching hospitals: the developing landscape
.
Acad Med
.
2008
;
83
(
6
):
541
549
.
3
Greysen
SR,
Richards
AK,
Coupet
S,
et al.
Global health experiences of US physicians: a mixed methods survey of clinician-researchers and health policy leaders
.
Global Health
.
2013
;
9
:
19
.
4
Nelson
BD,
Kasper
J,
Hibberd
PL,
et al.
Developing a career in global health: considerations for physicians-in-training and academic mentors
.
J Grad Med Educ
.
2012
;
4
(
3
):
301
306
.
5
Calhoun
JG,
Spencer
HC,
Buekens
P.
Competencies for global health graduate education
.
Infect Dis Clin North Am
.
2011
;
25
(
3
):
575
592
,
viii
.
6
Ablah
E,
Biberman
DA,
Weist
EM,
et al.
Improving global health education: development of a Global Health Competency Model
.
Am J Trop Med Hyg
.
2014
;
90
(
3
):
560
565
.
7
Battat
R,
Seidman
G,
Chadi
N,
et al.
Global health competencies and approaches in medical education: a literature review
.
BMC Med Educ
.
2010
;
10
:
94
.
8
Scott
J,
Revera Morales
D,
McRitchie
A,
et al.
Non-technical skills and health care provision in low- and middle-income countries: a systematic review
.
Med Educ
.
2016
;
50
(
4
):
441
455
.
9
Hagopian
A,
Spigner
C,
Gorstein
JL,
et al.
Developing competencies for a graduate school curriculum in international health
.
Public Health Rep
.
2008
;
123
(
3
):
408
414
.
10
Houpt
ER,
Pearson
RD,
Hall
TL.
Three domains of competency in global health education: recommendations for all medical students
.
Acad Med
.
2007
;
82
(
3
):
222
225
.
11
Redwood-Campbell
L,
Pakes
B,
Rouleau
K,
et al.
Developing a curriculum framework for global health in family medicine: emerging principles, competencies, and educational approaches
.
BMC Med Educ
.
2011
;
11
:
46
.
12
Sanders
D,
Haines
A.
Implementation research is needed to achieve international health goals
.
PLoS Med
.
2006
;
3
(
6
):
e186
.
13
Sastry
A.
Biggest obstacles to decent health care in the developing world are managerial
.
Huffington Post
.
July 26, 2013; updated September 25, 2013.
.
14
Ruxin
J.
Doctors without orders
.
Democracy
:
A Journal of Ideas
.
Summer 2008. No. 9. http://democracyjournal.org/magazine/9/doctors-without-orders. Accessed May 10, 2017
.
15
Zink
T,
Solberg
E.
Development of a global health curriculum for family medicine based on ACGME competencies
.
Teach Learn Med
.
2014
;
26
(
2
):
174
183
.
16
Kamaka
ML.
Designing a cultural competency curriculum: asking the stakeholders
.
Hawaii Med J
.
2010
;
69
(
6 suppl 3
):
31
34
.
17
Furin
J,
Farmer
P,
Wolf
M,
et al.
A novel training model to address health problems in poor and underserved populations
.
J Health Care Poor Underserved
.
2006
;
17
(
1
):
17
24
.
18
Hughes
R,
Huby
M.
The construction and interpretation of vignettes in social research
.
Soc Work Soc Sci Rev
.
2004
;
11
(
1
):
36
51
. .
19
Tashakkori
AM,
Teddlie
CB,
eds
.
SAGE Handbook of Mixed Methods in Social & Behavioral Research
.
Thousand Oaks, CA
:
SAGE Publications Inc;
2002
.
20
Witzel
A.
The problem-centered interview
.
Forum Qual Soc Res
.
2000:1(1):22.
.
21
Arthur
MA,
Battat
R,
Brewer
TF.
Teaching the basics: core competencies in global health
.
Infect Dis Clin North Am
.
2011
;
25
(
2
):
347
358
.
22
Viera
AJ,
Garrett
JM.
Understanding interobserver agreement: the kappa statistic
.
Fam Med
.
2005
;
37
(
5
):
360
363
.
23
Brewer
TF,
Saba
N,
Clair
V.
From boutique to basic: a call for standardised medical education in global health
.
Med Educ
.
2009
;
43
(
10
):
930
933
.
24
Moniz
T,
Arntfield
S,
Miller
K,
et al.
Considerations in the use of reflective writing for student assessment: issues of reliability and validity
.
Med Educ
.
2015
;
49
(
9
):
901
908
.
25
Lohfeld
L,
Goldie
J,
Schwartz
L,
et al.
Testing the validity of a scenario-based questionnaire to assess the ethical sensitivity of undergraduate medical students
.
Med Teach
.
2012
;
34
(
8
):
635
642
.

Author notes

Editor's Note: The online version of this article contains the structured interview tool.

Funding: This research was funded by the Medical Education Research Grant, Office of Graduate Medical Education, Center of Expertise in Medical Education, Partners Healthcare.

Competing Interests

Conflict of interest: The authors declare they have no competing interests.

Pilot results for some of the interviews were presented as a poster at the 6th Annual Consortium of Universities for Global Health Conference, Boston, Massachusetts, March 25–28, 2015.

The authors would like to express their gratitude to all interviewees who participated in this study, and for their valuable feedback.

Supplementary data