ABSTRACT

Background 

Psychological safety (PS) is the perception that it is safe to take interpersonal risks in the work environment. In teaching hospitals, PS may influence the clinical learning environment for trainees.

Objective 

We assessed whether resident physicians believe they are psychologically safe, and if PS is associated with how they rate satisfaction with their clinical learning experience.

Methods 

Data were extracted from the Learners' Perceptions Survey (LPS) of residents who rotated through a Department of Veterans Affairs health care facility for academic years 2011–2014. Predictors of PS and its association with resident satisfaction were adjusted to account for confounding and response rate biases using generalized linear models.

Results 

The 13 044 respondents who completed the LPS (30% response rate) were comparable to nonpediatric, non–obstetrics-gynecology residents enrolled in US residency programs. Among respondents, 11 599 (89%) agreed that “. . . members of the clinical team of which I was part are able to bring up problems and tough issues.” Residents were more likely to report PS if they were male, were in a less complex clinical facility, in an other medicine or psychiatry specialty, or cared for patients who were aged, had multiple illnesses, or had social supports. Nonpsychiatric residents felt safer when treating patients with no concurrent mental health diagnoses. PS was strongly associated with how residents rated their satisfaction across 4 domains of their clinical learning experience (P < .001).

Conclusions 

PS appears to be an important factor in resident satisfaction across 4 domains that evaluators of graduate medical education programs should consider when assessing clinical learning experiences.

What was known and gap

A safe, learning-oriented working environment is important to meet physician education objectives and ensure safe care in teaching settings.

What is new

Responses to a question about psychological safety from the Veterans Affairs Learners' Perceptions Survey were associated with how residents rated their satisfaction across 4 domains of the clinical learning experience.

Limitations

Limited response rate; psychological safety was measured using a single question.

Bottom line

Psychological safety is an important factor in resident satisfaction with their learning environment that can be measured by teaching institutions.

Introduction

The provisions of a learning-oriented culture and a productive work environment are important to the achievement of resident competencies.1  The Accreditation Council for Graduate Medical Education (ACGME) requires graduate medical education (GME) programs to provide a clinical environment where residents learn competencies for independent practice, and patients receive supervised, safe, and high-quality care.2 

A critical element often overlooked in assessing clinical learning environments is whether residents believe their clinical environment is psychologically safe.3  Psychological safety (PS) refers to the perception that there are no negative consequences to self, status, or career for taking interpersonal risks, such as reporting mistakes or problems.4  PS has been shown to be highly related to increased team engagement, increased rates of medical error reporting, personal work engagement, and quality system improvement.510 

While qualitative research has shown associations between communication and PS,9  few studies have measured resident perceptions of PS at their teaching hospitals and made correlations with attributes of their learning environment. This study applied advanced statistical methods to nationwide survey data to investigate the predictors of PS and its effects on resident satisfaction with their clinical facilities.

Methods

Setting

Since 1946, health professions education has been a statutory mission for the Department of Veterans Affairs (VA) health care facilities.11  Resident physicians train at VA medical centers where interprofessional team and patient-centered care are emphasized through the VA's Patient Aligned Care Team models of care.12 

Survey

Data came from the VA's Learners' Perceptions Survey (LPS),13,14  which is an anonymous, voluntary, computer-assisted, web-based survey of all students and residents engaged in supervised patient care at VA medical facilities as part of an accredited health professions education program. The purpose of the LPS is to assess VA performance in providing clinical training environments for health professions trainees. Since 2001, the survey has been administered annually by the Office of Academic Affiliations (OAA) using a data accounting system to ensure anonymity.15  The OAA works with local education leadership to send direct e-mails to potential eligible respondents to complete the survey.

In addition to trainee background information, the LPS contains 84 satisfaction-based questions across 9 domains describing a trainee's VA teaching, working, and clinical experiences for a given academic year. Respondents are asked a summary question at the end of each domain to describe their overall satisfaction with the domain. Satisfaction is reported on 5-point Likert scales (1, very dissatisfied, to 5, very satisfied). PS was measured using a 5-point Likert scale that asked whether a respondent agreed with the statement, “Members of the clinical team of which I was a part are able to bring up problems and tough issues.”9 

Survey responses show overall good internal consistency and 1-dimensional ratings by satisfaction domain,14  across disciplines,16,17  with good predictive,18  and construct validity.19 

Data

In this study, LPS survey data were limited to resident physicians who completed the survey from July 1, 2010, through June 30, 2014. Domain satisfaction was scored for each respondent as a 5-point ordinal scale computed from responses to the domain summary question, and as a continuous measure calculated as z scores computed from responses to 4 domain element questions. Domain z scores are sufficient statistics that represent satisfaction intensities as latent factors.20  The 4 domains were learning, faculty and preceptors, personal, and systems and processes for handling medical errors. These domains were selected to represent the broad range of learning experiences.14 

Covariates to predict PS included the respondent's self-reported sex; academic level (eg, postgraduate year 1 [PGY-1], PGY-2, PGY-3, etc); specialty as defined by ACGME21  (table 1 footnote); medical school status (US versus non-US graduate); characteristics of patients seen based on percentage of patients who are older than 65 years, chronic medical illness, multiple medical illnesses, mental illness, alcohol/substance dependence, low socioeconomic status, and lack of social support; and hospital service complexity. The VA classifies service complexity on a 5-point Likert scale based on the facility's patient volume, types of services offered, patient complexity, teaching status, number of training programs, and research activities (table 1 footnote).

table 1

Characteristics of Resident Physician Sample

Characteristics of Resident Physician Sample
Characteristics of Resident Physician Sample

Respondents often have different thresholds when mapping the intensity of their satisfaction on a 5-point Likert scale. Therefore, we computed a response bias index to measure the extent to which responders give higher or lower satisfaction ratings than their colleagues. The index was computed for each respondent as a z score; this was calculated from satisfaction responses to 3 element questions describing the facility's computerized medical record system, location convenience, and parking availability, minus the mean z score across all respondents at the same facility and academic year. Response bias indices are positive, zero, or negative depending on whether a respondent gave a higher, equivalent, or lower satisfaction rating, respectively, than their peers.

The collection of LPS survey data, processing, and analyses were conducted under license from the Office of Management and Budget (Information Collection No. 2900-0691, VA Form No. 10-0439). Data use was covered under the US Office of Management and Budget (No. 2900-0691) and is exempt from human subjects' research.

Data Analyses

Analyses to compute associations between PS, its predictors, and its satisfaction outcomes were done for both ordinal and continuous satisfaction scores, and to account for observed and unobserved confounding covariates. For part 1, observed covariates were examined as independent predictors of PS by fitting generalized linear models22  (GLMs) with cumulative logit link functions and multinomial distributions. For part 2, associations between PS and domain satisfaction were computed using a GLM for 5-point ordinal domain, and for continuous z score domain using a GLM with identity link function and Gaussian or gamma distribution. To control for confounding when computing associations between PS and domain satisfaction, we used a control function technique commonly utilized by social scientists.23  This allows estimates of the association between PS and domain satisfaction to be (1) adjusted for known, observed covariates that can confound the association; (2) corrected for the response bias index to account for respondents having different reporting thresholds; and (3) calibrated using the residual inclusion method24  to control for unobserved confounders that were not available in the study data set.

Results

Sample Characteristics

Of a total of 43 000 residents with known e-mail addresses, 13 044 completed the LPS (30% completion rate). The LPS resident physician sample has been shown to be comparable to the distribution of ACGME residents (exclusive of obstetrics-gynecology and pediatrics) with regard to specialty choices, academic level, US graduate status, geographic area, and sex.25 

Description of the study sample is provided in table 1. Most respondents came from facilities within the top 3 of the 5 VA complexity levels (n = 11 480, 88%), and 48% (n = 6316) of respondents were from internal medicine and its subspecialties. Most residents were in their first 3 years of training (n = 9087, 70%), US graduates (n = 9713, 75%), and male (n = 7418, 57%). Most respondents reported seeing 25% or more patients who were 65 years or older (91%, 11 754 of 12 910), had a medical illness (95%, 12 310 of 12 909), or had multiple illnesses (96%, 12 326 of 12 910). On the other hand, only 67% reported seeing 25% or more patients with mental illness (8675 of 12 909) or who lacked social support (8629 of 12 909). Approximately 89% (11 599 of 13 044) of the sample agreed their VA experience was psychologically safe, with approximately half (48%, 6204) strongly agreeing.

Psychometric Properties of LPS

Element responses by domain were shown to be reliable (intraclass correlations between 0.57 and 0.87), consistent (Cronbach's alpha from 0.93 to 0.98), and 1-dimensional (second-largest eigenvalue from 0.26 to 0.92). The 80% intraclass correlation for all 41 elements across the 4 domains indicates responses clustered around their respective domain relative to the total variance across all element questions. PS responses did not cluster by facility, with between-facility responses accounting for only 12% of the total variance among respondents. This underscores the importance of analyzing PS at the respondent level, rather than at a facility-aggregated level. Responses to the single PS question explained 88.4% of the total variance to the 2 related questions: “I feel free to question the decisions or actions of those with more authority” (asked in 2011), and “It is safe to take a risk in this clinical team” (asked in 2012–2014).

Predictors of PS

Overall, 11 599 respondents (89%) agreed that members of the clinical team could “bring up problems and tough issues,” including 6204 (48%) who had strongly agreed. Table 2 reports how factors independently were associated with PS. Responders who reported PS tended to be in less complex facilities, in other medicine and psychiatry specialties, male, and saw more patients who were aged, had multiple illnesses, with positive social supports, and had no co-occurring mental illness. PS was also associated with responders who tended to over report satisfaction (ie, positive response bias). No association was found between PS and academic level.

table 2

Independent Associationsa Between Selected Factors and Residents' Psychological Safety

Independent Associationsa Between Selected Factors and Residents' Psychological Safety
Independent Associationsa Between Selected Factors and Residents' Psychological Safety

Association Between PS and Domain Satisfaction

Estimates of the association between PS and domain satisfaction are reported in tables 3 and 4, by analytic strategy (adjusted, corrected, and calibrated) and domain scale (5-point Likert ordinal and continuous z scores). Overall, PS was strongly associated with residents' satisfaction of their clinical learning experiences, with respondents who reported 1 higher level of PS being 5.7 times more likely to report a higher level of satisfaction for their clinical learning environment (95% confidence interval [CI] 5.4–6.1, P < .001). The odds ratio drops to 5.6 (95% CI 5.3–6.0) when adjusting for observed covariates, to 4.2 (95% CI 4.0–4.5) when also correcting for response biases, and to 3.0 (95% CI 2.8–3.3, P < .001) when calibrating for unobserved confounding factors. For z score satisfaction ratings, 1 higher level of PS was associated with a Cohen's d of 1.3 standard deviates (95% CI 1.2–1.5, P < .001). Final PS effect size estimates varied only slightly by domain.

table 3

Association Between Feeling Psychologically Safe and Satisfaction With Clinical Learning Experiences, by Domain and Estimation Method

Association Between Feeling Psychologically Safe and Satisfaction With Clinical Learning Experiences, by Domain and Estimation Method
Association Between Feeling Psychologically Safe and Satisfaction With Clinical Learning Experiences, by Domain and Estimation Method
table 4

Controlled Associationsa Between Feeling Psychologically Safe and Satisfaction With Clinical Learning Experience, by Domain and Method of Computing Satisfaction

Controlled Associationsa Between Feeling Psychologically Safe and Satisfaction With Clinical Learning Experience, by Domain and Method of Computing Satisfaction
Controlled Associationsa Between Feeling Psychologically Safe and Satisfaction With Clinical Learning Experience, by Domain and Method of Computing Satisfaction

Discussion

This study applied advanced analytic methods to a national data set to provide quantitative evidence that trainees overall gave high ratings for psychological safety in the VA health care system. The study also showed that PS ratings varied by specialty, clinical setting, and resident, even within the same facility and time period. Finally, despite different analytic (adjusted, corrected, calibrated) and scoring (ordinal, continuous) methods, PS had a strong, statistically significant, and robust association with how residents rated satisfaction with their VA learning, personal, and clinical experiences.

Specifically, our study showed that psychiatry and specialties falling under the “other medicine” category (eg, allergy and immunology, dermatology, family medicine, neurology, physical medicine and rehabilitation) had higher levels of PS (table 2). It is possible that this greater sense of PS among these specialties depends not only on the residents' interaction with the clinical environment but may also reflect an inherent predisposition to PS, as noted by a higher tolerance to ambiguity for those choosing the field of psychiatry.26,27 

The large PS effect on satisfaction with personal experiences is consistent with qualitative studies that have linked PS to workplace communications in the VA.8  The ACGME Resident Survey revealed that resident-faculty interactions, procedural volume, variety, and didactic offerings were among the attributes residents used to gauge training quality.28  These findings underscore the importance of psychologically safe clinical settings if residents are to achieve ACGME milestones,29  provide high-quality patient care,30  and foster professionalism and accountability. These, in turn, are associated with resident competence31  and allow clinical obligations to be resolved without fear of intimidation.32 

Our finding that PS is more prevalent in less complex facilities supports the theory that complex settings present higher levels of stress by requiring navigation through more difficult systems. It has been suggested that in complex systems, education should focus on skills learned at the time when mistakes are made.33  GME-adapted traditional apprenticeship models are based on “legitimate peripheral participation,”34  which focuses on a resident's evolution of professional identity—from being a largely passive participant in low-risk situations when a master practitioner is nearby to an active one who has integrated and is demonstrating the knowledge, skills, and attitudes of the community. Migration theory35  contextualizes this further by assuming that demanding work systems tolerate margins of acceptable performance (eg, clinical teams under pressure to provide expeditious care may practice at a “normal-illegal stage” where minor infringements of safety are implicitly accepted).36 

Contrary to popular notions that junior residents perceive less PS than their senior counterparts, we found no association between PS and academic level after adjusting for confounding factors and utilizing a response bias index. Our finding that men report greater PS is consistent with women reporting greater anxiety, less self-confidence, and more apprehension to communicate with their supervising physicians.37,38 

More research is needed to understand where boundaries lie between safety and harm when residents ask for help. For instance, attending physicians describe “strong” residents as being able to handle heavy workloads without supervision.39  Residents' failure to ask for help may stem from concerns about appearing “weak,” or a perceived erosion of their autonomy and professional identity. Conversely, supervisor approachability and scope of the clinical question are also determinants of trainee decisions to seek supervision.34,40  Similar attributes are drivers of PS in other industries, leading to job satisfaction, organizational commitment, and quality improvement work.41 

The study has limitations. First, data were based on self-reports. Second, PS was measured based on a single question that was available across all 4 study years. Third, surveys may not be the best vehicle to assess PS, as what residents say they will do and what they will actually do are not always congruent, especially given the external pressures of medicine.4244  A fourth limitation is the low response rate of 30%. However, our purpose was not to infer the prevalence of psychological safety, but rather to make more robust comparisons in how PS, PS predictors, and satisfaction outcomes co-vary. There were no significant differential response rates among respondents in psychologically safe and unsafe environments that could bias association estimates. A fifth limitation is that the same resident physician may have responded to the LPS survey in different years. Finally, we used an observational data set to study PS, drawing inferences from nonrandomized concurrent comparison groups with blinded participants' study, where issues of confounders and selection biases can be addressed analytically.45,46 

Future studies should assess organizational structure and PS in medical education and health care settings, and could use more direct methods of program evaluation.

Conclusion

Overall, this study provides strong evidence that PS is important to residents' perceptions of their clinical learning experience. Organizations that promote and measure psychologically safe learning environments may increase resident satisfaction with their GME experiences.

References

References
1
Hoff
TJ,
Pohl
H,
Bartfield
J.
Creating a learning environment to produce competent residents: the roles of culture and context
.
Acad Med
.
2004
;
79
(
6
):
532
539
.
2
Accreditation for Graduate Medical Education
.
CLER Pathways to Excellence: expectations for an optimal clinical learning environment to achieve safe and quality patient care
. ,
2016
.
3
Colbert-Getz
JM,
Kim
S,
Goode
VH,
et al.
Assessing medical students' and residents' perceptions of the learning environment: exploring validity evidence for the interpretation of scores from existing tools
.
Acad Med
.
2014
;
89
(
12
):
1687
1693
.
4
Edmondson
A.
Psychological safety and learning behavior in work teams
.
Admin Sci Q
.
1999
;
44
(
2
):
350
383
.
5
Edmondson
AC.
Managing the risk of learning: psychological safety in work teams
.
In
:
West
MA,
Tjosvold
D,
Smith
KG,
eds
.
International Handbook of Organizational Teamwork
.
2016
.
6
Carmeli
A,
Gittell
JH.
High-quality relationships, psychological safety, and learning from failure in work organizations
.
J Organ Behav
.
2009
;
30
(
6
):
709
729
.
7
Carmeli
A,
Brueller
D,
Dutton
JE.
Learning behaviours in the workplace: the role of high-quality interpersonal relationships and psychological safety
.
Syst Res Behav Sci
.
2009
;
26
(
1
):
81
98
.
8
Naveh
E,
Katz-Navon
T.
Antecedents of willingness to report medical treatment errors in health care organizations: a multilevel theoretical framework
.
Health Care Manage Rev
.
2014
;
39
(
1
):
21
30
.
9
Yanchus
NJ,
Derickson
R,
Moore
SC,
et al.
Communication and psychological safety in veterans health administration work environments
.
J Health Organ Manag
.
2014
;
28
(
6
):
754
776
.
10
Aranzamendez
G,
James
D,
Toms
R.
Finding antecedents of psychological safety: a step toward quality improvement
.
Nurs Forum
.
2015
;
50
(
3
):
171
178
.
11
Gilman
SC,
Chang
BK,
Zeiss
RA,
et al.
The academic mission of the Department of Veterans Affairs
.
In
:
TW
Miller,
ed
.
The Praeger Handbook of Veterans' Health: History, Challenges, Issues, and Developments. Volume 1: History, Eras, and Global Healthcare
.
Santa Barbara, CA
:
Praeger;
2012
:
53
82
.
12
Rosland
AM,
Nelson
K,
Sun
H,
et al.
The patient-centered medical home in the Veterans Health Administration
.
Am J Manag Care
.
2013
;
19
(
7
):
e263
e272
.
13
Keitz
SA,
Holland
GJ,
Melander
EH,
et al.
The Veterans Affairs Learners' Perceptions Survey: the foundation for education quality improvement
.
Acad Med
.
2003
;
78
(
9
):
910
917
.
14
Kashner
TM,
Bernett
DS,
Wicker
AB.
Learners' Perceptions Survey: instructions manual for data users
.
Washington, DC: Office of Academic Affiliations; 2015.
,
2016
.
15
Kashner
TM,
Hinson
RS,
Holland
GJ,
et al.
A data accounting system for clinical investigators
.
J Am Med Inform Assoc
.
2007
;
14
(
4
):
394
396
.
16
Cannon
GW,
Keitz
SA,
Holland
GJ,
et al.
Factors determining medical student and resident satisfaction during VA-based training: results from the VA Learners' Perceptions Survey
.
Acad Med
.
2008
;
83
(
6
):
611
620
.
17
Kaminetzky
CP,
Keitz
SA,
Kashner
TM,
et al.
Training satisfaction for subspecialty fellows in internal medicine: findings from the Veterans Affairs (VA) Learners' Perceptions Survey
.
BMC Med Educ
.
2011
;
11
:
21
.
18
Lam
HT,
O'Toole
TG,
Arola
PE,
et al.
Factors associated with the satisfaction of millennial-generation dental residents with their training experience
.
J Dent Educ
.
2012
;
76
(
11
):
1416
1426
.
19
Kashner
TM,
Henley
SS,
Golden
RM,
et al.
Studying effects of ACGME duty hours limits on resident satisfaction: results from the VA Learners' Perceptions Survey
.
Acad Med
.
2010
;
85
(
7
):
1130
1139
.
20
Hardouin
JB,
Mesbah
M.
Clustering binary variables in subscales using an extended Rasch model and Akaike information criterion
.
Comm Stats Theory Methods
.
2004
;
33
(
6
):
1227
1294
.
21
Accreditation Council for Graduate Medical Education
.
ACGME data resource book: academic year 2012–2013
. ,
2016
.
22
McCullagh
P,
Nelder
JA.
Generalized Linear Models
.
New York, NY
:
Chapman and Hall;
1989
.
23
Heckman
J,
Navarro-Lozano
S.
Using matching, instrumental variables, and control functions to estimate economic choice models
.
Rev Econ Stat
.
2004
;
86
(
1
):
30
57
.
24
Terza
JV,
Basu
A,
Rathouz
PJ.
Two-stage residual inclusion estimation: addressing endogeneity in health econometric modeling
.
J Health Econ
.
2008
;
27
(
3
):
531
543
.
25
Kashner
TM,
Hettler
DL,
Zeiss
RA,
et al.
Has interprofessional education changed learning preferences? A national perspective
.
Health Serv Res
.
2016
Mar
18
.
Epub ahead of print.
doi: .
26
Lally
J,
Cantillon
P.
Uncertainty and ambiguity and their association with psychological distress in medical students
.
Acad Psychiatry
.
2014
;
38
(
3
):
339
344
.
27
Geller
G,
Faden
RR,
Levine
DM.
Tolerance for ambiguity among medical students: implications for their selection, training and practice
.
Soc Sci Med
.
1990
;
31
(
5
):
619
624
.
28
Philibert
I.
Satisfiers and hygiene factors: residents' perceptions of strengths and limitations of their learning environment
.
J Grad Med Educ
.
2012
;
4
(
1
):
122
127
.
29
Philibert
I,
Brigham
T,
Flynn
TC.
The next GME accreditation system—rationale and benefits
.
N Engl J Med
.
2012
;
366
(
1
):
1051
1056
.
30
Institute of Medicine
.
Crossing the quality chasm: a new health system for the 21st century
.
Washington, DC: National Academy of Sciences; 2001.
,
2016
.
31
Gillespie
C,
Paik
S,
Ark
T,
et al.
Residents' perceptions of their own professionalism and the professionalism of their learning environment
.
J Grad Med Educ
.
2009
;
1
(
2
):
208
215
.
32
Holt
KD,
Miller
RS,
Philibert
I,
et al.
Residents' perspectives on the learning environment: data from the accreditation council for graduate medical education resident survey
.
Acad Med
.
2010
;
85
(
3
):
512
518
.
33
Edmondson
AC.
Learning from mistakes is easier said than done: group and organizational influences on the detection and correction of human error
.
J Appl Behav Sci
.
1996
;
32
(
1
):
5
28
.
34
Lave
J,
Wenger
E.
Situated Learning: Legitimate Peripheral Participation
.
Cambridge, MA
:
Cambridge University Press;
1991
.
35
Rasmussen
J.
Risk management in a dynamic society: a modeling problem
.
Safety Sci
.
1997
;
27
(
1–2
):
183
213
.
36
Amalberti
R,
Vincent
C,
Auray
Y,
et al.
Violations and migrations in health care: a framework for understanding and management
.
Qual Saf Health Care
.
2006
;
15
(
suppl 1
):
66
71
.
37
Blanch
DC,
Hall
JA,
Roter
DL,
et al.
Medical student gender and issues of confidence
.
Patient Educ Couns
.
2008
;
72
(
3
):
374
381
.
38
Babaria
P,
Adedin
S,
Nunez-Smith
M.
The effect of gender on the clinical clerkship experience of female medical students: results from a qualitative study
.
Acad Med
.
2009
;
84
(
7
):
859
866
.
39
Kennedy
TJ,
Regehr
G,
Baker
GR,
et al.
“It's a cultural expectation…”: the pressure on medical trainees to work independently in clinical practice
.
Med Educ
.
2009
;
43
(
7
):
645
653
.
40
Kennedy
TJ,
Regehr
G,
Baker
GR,
et al.
Preserving professional credibility: grounded theory study of medical trainees' requests for clinical support
.
BMJ
.
2009
;
338
:
b128
.
41
Seibert
SE,
Wang
G,
Courtright
SH.
Antecedents and consequences of psychological and team empowerment: a meta-analytic review
.
J Appl Psychol
.
2011
;
96
(
5
):
981
1003
.
42
Farnan
JM,
Johnson
JK,
Meltzer
DO,
et al.
Resident uncertainty in clinical decision making and impact on patient care: a qualitative study
.
Qual Saf Health Care
.
2008
;
17
(
2
):
122
126
.
43
Bush
RW.
Supervision in medical education: logical fallacies and clear choices
.
J Grad Med Educ
.
2010
;
2(1);141–143.
44
Loo
L,
Puri
N,
Kim
DI,
et al.
“Page me if you need me”: the hidden curriculum of attending-resident communication
.
J Grad Med Educ
.
2012
;
4
(
3
):
340
345
.
45
Sullivan
GM.
Getting off the “gold standard”: randomized controlled trials and education research
.
J Grad Med Educ
.
2011
;
3
(
3
):
285
289
.
46
Kashner
TM.
How good are observational datasets in assessing psychiatric treatment outcomes?
Am J Psych
.
2012
;
169
(
3
):
244
247
.

Author notes

Funding: The authors report no external funding source for this study.

Competing Interests

Conflict of interest: The authors declare they have no competing interests.

The authors would like to thank the Office of Academic Affiliations (OAA); the OAA National Evaluation Workgroup (Sheri A. Keitz, MD, PhD, Chair); Christopher T. Clarke, PhD, Terry V. Kruzan, George E. McKay, David S. Bernett, and Laura Stefanowycz, OAA Data Management and Support Center; and the network of designated education officers and associated chiefs of staff for education at VA medical centers.