Toward Defining and Measuring Social Accountability in Graduate Medical Education: A Stakeholder Study

Public financing of graduate medical education (GME) began in 1965 with the passage of Medicare. The intent was for GME to create a future physician workforce and provide services to underserved populations in exchange for taxpayer support.1–3 

Since 1965, Medicare has remained the main source of GME funding in the United States, now spending more than $9.5 billion annually.4 Increased spending has run parallel to increased calls for financial and social accountability in GME.5–7 For example, the Council on Graduate Medical Education (COGME), the American Medical Association, the Association of American Medical Colleges, and the Institute of Medicine, have all called for significant changes within GME, addressing issues related to financial and social accountability (eg, cost containment, health disparities, and primary care shortages).8–11 The Medicare Payment Advisory Committee (MedPAC) has gone further, recommending the creation of an advisory body to develop performance measures and increased accountability linked to GME payment.12,13 The Patient Protection and Affordable Care Act gave related responsibility to COGME to “develop, publish, and implement performance measures; develop and publish guidelines for longitudinal evaluations; and recommend appropriation levels” for GME programs based on an accountability framework.14 Further, the Institute of Medicine is currently hosting a study of GME accountability.15 

In spite of such calls for greater accountability in GME, a precise definition of accountability and the specifics for measuring it remain elusive. To begin to understand relevant perspectives around these issues, we purposefully sought out GME stakeholders with expertise in GME to explore their definitions of “social accountability” with respect to GME, their suggestions for measuring social accountability, and what they think the perceived value of doing so is.

Based on a review of prior literature regarding social accountability16 and other relevant citations from that literature,1–10,12–14,17–19 the research team developed a draft interview guide to address our study aims. Specifically, we sought to examine definitions of social accountability in GME, as well as suggestions for, perceived value in, and barriers to measurement. The lead investigator (A.T.R.) created the initial interview guide, and other team members contributed to iterative revisions. The guide was piloted with 11 GME stakeholders (representatives from GME training sites, governmental bodies, and health care organizations and associations), allowing researchers to optimize prompts and probes and refine specific questions.20 We did not record pilot interviews or include data from these sessions in our final analysis because of the lack of standardization.

We used a purposive sampling strategy.21,22 Study investigators began by developing a GME stakeholder list. The list included organizations with experience in providing GME or knowledge about the policy and history of GME that would understand the implications of GME policy changes. On the list were members of GME training programs, government agencies, medical associations, and health care organizations. Investigators were careful to include groups that might be reticent about GME measurement, those that might have unconventional ideals about accountability, and those that could benefit from not being held socially accountable. Working from the list, investigators then reached out to individuals at various levels in all groups on the brainstormed list (using established contacts, leads from Internet sites, and direct referrals) to increase the likelihood of a full range of perspectives. In clinical organizations, we drew from both generalists and specialists.

Of 38 individuals we initially invited to participate via e-mail, 18 did not respond, 3 declined, and 1 canceled a scheduled interview and did not reschedule. Those who declined or canceled cited time limitations or inadequate knowledge base as reasons for not participating. Nonparticipants did not differ meaningfully from participants in terms of organization, job title, or medical specialty.

In addition to purposive sampling, we remained open to identifying other critical perspectives via snowball methodology: individuals who participated in interviews referred the study team to other potential interviewees.22,23 Ultimately, 2 additional interviews resulted from 10 referrals.

We conducted 18 interviews between January and April 2011 (table 1). Despite being a relatively small sample from a broad set of stakeholders, the range of ideas about GME accountability converged rapidly, suggesting saturation,22 or that few new ideas or themes were likely to emerge from continued sampling within this frame.

The 2 lead investigators who conducted all interviews (A.T.R. and S.A.L.) underwent intensive training in interviewing techniques by an experienced anthropologist and qualitative researcher. Each investigator observed pilot interviews by the other investigator to standardize technique and ensure consistency. Both investigators debriefed after pilot interviews with the consulting anthropologist, who observed the interviews and advised on improving technique and consistency in information gathering.

Study interviews occurred in person (n  =  6) or by phone (n  =  12). All participants provided verbal informed consent. Interviews were semistructured, following the interview guide, but participants were encouraged to deviate from the questions where appropriate to explore related ideas. Interviews ranged from 30 to 90 minutes and were audiorecorded. Interviewer notes and verbatim transcriptions comprised our primary data, all of which were deidentified.

Data analysis was iterative and continuous.22 The 2 interviewers independently reviewed audiorecordings and field notes after interviews, then convened to develop an iteratively defined codebook, identifying frequently espoused concepts and perspectives. Review of subsequent interview recordings and notes generated new codes, which the team then applied to earlier interviews. Analysis started with the first interview and continued through iterative revisions to the last. Two investigators independently applied codes to each interview, and investigators resolved any disagreement through discussion and reimmersion into the text where necessary.22 Analysis involved the broader team of investigators to review and corroborate codes, offer alternative codes for given segments of text, and achieve consensus around emerging themes.22 All investigators contributed to the identification of patterns within the coded text, resulting in the themes shown in table 2.

The American Academy of Family Physicians' Institutional Review Board approved the study protocol.

Of the 18 participants, 17 were physicians and 10 were primary care physicians; 6 of the 18 were women. Participants included medical school administrators, department chairs, residency program directors, and members of academic health care organizations, federal advisory committees/commissions, and accreditation organizations.

In defining social accountability, 3 themes emerged:

A recurrent idea was that institutions should address the workforce needs of the community and/or nation (eg, “Getting out of our public investment what we need from our trainees [for a] workforce of the future”). Opinions varied about what workforce needs exist. Some participants cited primary care shortages and others noted health professional shortage areas as indicators of workforce deficits. Participants pointed out that different geographic regions of the country have different specialty needs. One participant said that GME should produce “a workforce that is diverse—covering all primary care and specialty needs—that will serve geographic areas that are underserved.”

Participants mentioned educating physicians to provide high-quality care as a key accountability measure: “Society deserves to have the best trained physicians possible. It is the responsibility of GME institutions to do whatever it takes to optimize the quality of their trainees.” Other suggestions highlighted quality of care as a training outcome, for example, stating that appropriate medical training should emphasize the “use of HIT [health information technology], training in teams… [and] other things…such as, ‘Are you training people around quality and safety; are you training them on how to look at a panel and improve outcomes for a population of patients?’”

A majority of respondents identified service as a key aspect of social accountability. Service was sometimes defined as individual patient care, but more often as work for the benefit of surrounding communities, geographic patient populations, or the nation. One participant said social accountability should include “an awareness on the part of the organization, about all stakeholders out there, not just the resident or patient being taken care of, but the entire community.” Another participant said GME institutions should have “a broader social mission to care for the underserved and vulnerable populations in addition to the responsibility to care for individual patients.”

In terms of measuring social accountability in GME, participants talked about 3 themes related to value, implementation, and barriers.

Most participants thought GME institutions should be socially accountable. Most believe there is value in measuring whether or not the GME system successfully trains a physician workforce to serve society's needs. There was also enthusiasm around the value of measurement to actively push GME stakeholders to strive for greater accountability. Some thought the value lay in increasing residents' awareness of population medical needs, enhancing cost-effectiveness, and creating better relationships with communities. Participants also believed “[measuring social accountability] will show future physicians that [social accountability] is something that is valuable. If it is measured, promoted, and used as a selling point, it may incite the interest of future physicians.”

Many participants stated that trainees' ultimate specialties and scope of practice should be a measure: “You could measure whether people are going into the needed professions…see what they are doing 1, 3, 5 years after so-called primary care training.” Some suggested other quantitative measures, for example, “Those characteristics that allow for measurement of outcomes that can be used as proxies for social accountability, including measurement of complications rates, lengths of stay, repeat admissions.” Several participants indicated that there should be community participation in institutional planning and evaluation, and that some measure of community impact would be important. Many participants suggested that accrediting bodies should play an important role, for instance by incorporating social accountability in accrediting requirements (eg, “Accrediting bodies should set some standards and be a warehouse of best practices and materials and could say, ‘I think you should go out and do this and we are going to hold you accountable.’”). Funding was frequently mentioned as a catalyst for making measurement a reality (eg, “Maybe programs that train more primary care residents should get more GME reimbursement.”).

Participants noted that there are currently no financial incentives to change: “I don't see why professional institutions would change…I don't know if there is a pressure to change, unless you changed the funding of GME to be based on social accountability.” Additionally, there was concern that aligning financial incentives with social accountability would meet with resistance: “I am not sure you are going to get buy-in from teaching hospitals, since this could imply that their funding could get diminished.” Another noted, “When you talk about $9 billion, it means there are a lot of fingers in the pot, and a lot of people who don't want to change things.”

Participants also observed the entrenched complexity of current GME payments as a barrier to meaningful funding changes: “You are getting to a higher level and that would involve Congress, MedPAC, Veterans Administration, and Department of Defense buy-in.” Some participants suggested specific GME incentives, reconfiguring “IME [indirect medical education] payments…to promote social accountability…but I don't know if Medicare will want to go there.”

Concerns arose about possible negative consequences of measuring social accountability. One participant stated, “the public does not understand scores, one [institution] may be ranked higher than another, with the public automatically assuming that one is better than another—that there is a difference, even though there may not be. This is why institutions get nervous about making their data public; it is toxic and defeats the purpose.” There was also concern that some desired outcomes, such as professionalism or cultural sensitivity, may not be measurable. This carried over to quantitative measures: “I think just counting bodies and just counting specialties is a bit of a flawed approach.”

Feasibility came up as an issue, given the number of institutions and programs to be measured: “I think it would be an enormous task and would not be very successful. You have over 8000 thousand programs out there and all you can look at is overall threads in terms of training and accountability.” Several participants thought institutional measures would be more appropriate than measures of program, and 1 participant suggested self-reporting to address feasibility issues.

Stakeholders almost universally expressed that GME institutions should be socially accountable and attentive to broader societal needs. Key themes in defining social accountability centered on creating an adequate workforce to address physician shortages, training in high-quality care for patients, and providing service to communities and the public. One participant captured this succinctly: “Look at the type of physician you are training, the environment they are trained in, and what they are doing for direct service in their environment.” Perhaps because participants were GME stakeholders, none questioned the value of public investment in physician education or its potential impact on the future physician workforce.

Our analysis suggests that many barriers may inhibit efforts to promote GME institutions' social accountability. This is an important finding, as prior work on social accountability has not explored implementation, including possible barriers to using measures of social accountability.

Notably, the United States is not alone in its new focus on social accountability for medical education.24,25 Canadian medical schools have adopted a social accountability mandate, using the World Health Organization's definition: “The obligation to direct [medical school] education, research, and service activities towards addressing the priority health concerns of the community, the region, and the nation's [medical schools] have a mandate to serve.”16,26 Canada developed several nationwide initiatives that partner academia with the community and the government to address the needs of the Canadian population.27,28 Even in Canada, however, individual institutions' responses to the social accountability movement have varied because of the broad framework used to define social accountability activities.27 

With increasing calls for social accountability for GME in the United States, Canada's collaborative approach—with leadership by academic institutions and key associations and support from government agencies—may provide a model.17 In the United States, commitment to social accountability for GME may be far more variable, but clear guidelines and financial incentives could be a catalyst for adoption. MedPAC suggested financial incentives as a means to accelerate efforts to improve accountability around quality of care and training and to improve the value of the health care delivery system.18,19 Furthermore, MedPAC's guidelines and incentives appear to be supported by the president's budget recommendations for 2013, which propose that those who receive GME payments meet specific value, quality, and primary care training standards.29 

Past evaluations of the Health Resources and Services Administration (HRSA) Title VII program, which funds efforts to increase primary care, care in underserved areas, the number of minority/disadvantaged students going into health professions, and the number of faculty in health care education, demonstrate that institutional programs can influence GME outcomes, even with relatively small investments.30,31 GME funding incentives tied to workforce needs could be 25 to 50 times larger than Title VII and significantly influence social accountability.32 HRSA is making additional strategic investments to expand training in underserved settings and to prepare faculty for teaching in new primary care clinical models.33 The evidence yielded by HRSA's investments could inform more accountable Medicare GME funding.

Potential limitations of our study include the small sample and restricted sample frame. We included a variety of major stakeholders in GME funding, representing a range of clinical specialties, as well as policy makers, advocates, accreditors, and educators, but did not interview patients, community leaders, community health care providers, business leaders, or insurers. We focused our exploration of GME social accountability on the stakeholders closest to the current policy discussions but recognize that other stakeholders may offer valuable and potentially differing input. We recognized our own potential biases as researchers—in particular a shared background in primary care—but actively sought out alternative points of view, making certain that interview participants with medical degrees represented multiple specialty areas. Strengths of our approach include the extensive pilot testing, purposive snowball sampling, involvement of experienced qualitative researchers, standardization of interviewing techniques, and rigorous analytic approach, which helped us achieve rapid thematic saturation, suggesting reasonable generalizability among included stakeholder groups.

GME stakeholders generally report that there would be value in measuring social accountability, and they offered suggestions for definitions and measurement, while at the same time recognizing there would be barriers and disincentives to implementation. To quote 1 participant: “We can't keep plodding along saying, ‘Keep giving us money, and we will come up with a better plan,’ because we are not getting there.” We believe future research should seek additional perspectives from other stakeholders, such as community and patient groups, and should seek to assess the degree to which measures may already be in place by broadly surveying program directors. By engaging all stakeholders in the development of socially accountable metrics, we may be able to meaningfully address increasing calls for GME financing reform and accountability.