A growing body of research has documented a decade-long decline in the scope of practice of family physicians,1–5 despite residency training designed to deliver high-quality care and resident intent to practice across the spectrum of family medicine, including pediatrics, geriatrics, obstetrics, and inpatient and ambulatory care.6 These findings raise questions about whether there is a lack of training, a lack of practice, or employer restrictions that limit opportunities for family physicians. Recent changes in the accreditation system for graduate medical education,7 including the unification of allopathic and osteopathic accreditation8 and the implementation of milestones,9 also raise questions on how these changes will affect graduates' practice. Across specialties, a dearth of longitudinal data spanning undergraduate medical education to practice after graduation presents a barrier to answering these questions.
The Accreditation Council for Graduate Medical Education (ACGME) requires family medicine residency programs to survey their graduates.10 Graduate surveys have assessed practice patterns,4,11–13 rural-urban and gender differences in procedures,14 declines in pregnancy care,5 and participation in community-related activities.15 Graduate survey data have also examined the effects of training on practice, such as the impact of reduced clinical and educational work hours and enhanced supervision requirements.16,17 To date, no systematically collected national data exist to contextualize residency outcomes in family medicine.
In 2014, the Association of Family Medicine Residency Directors (AFMRD) and the American Board of Family Medicine (ABFM) convened a steering committee to identify the priorities for a national graduate survey, with the idea of using data collected for ABFM business purposes to fulfill the ACGME graduate survey requirement.18 The committee included representatives from the AFMRD, ABFM, Council of Academic Family Medicine Educational Research Alliance, ACGME, MedEdNet practice-based research network, and a recent graduate. The committee solicited proposals for survey development, and 2 of the authors (A.K.H.W. and F.M.C.) were selected to create the National Family Medicine Graduate Survey. In this article, we describe the survey's creation, piloting, and validation so that other interested specialties may consider a similar approach.
Survey Development
The steering committee required the survey to (1) assess several specific elements considered critical; (2) exclude data that the ABFM already collected; and (3) take no more than 10 to 12 minutes to complete. ABFM Diplomates would complete the survey as part of their family medicine certification process, which takes place approximately 3 to 4 years after residency graduation. Aggregated reports by program would be provided to family medicine residencies.18
The project was determined to be survey development and, thus, exempt from Institutional Review Board review. The ABFM paid for the development of the survey, and bears the costs of implementation and maintenance.
Literature Review and Needs Assessment
We conducted an in-depth review of items in the ABFM's certification and recertification questionnaires as well as several existing family medicine graduate surveys. From this, we developed a list of possible topics and a first draft of the survey. The main topics identified were practice and work schedule; scope of practice; adequacy of training, competence, and practice of content areas/procedures; practice characteristics; satisfaction; professional activities; and patient characteristics.
This topic list and draft survey were reviewed by the members of the steering committee and representatives from the Society of Teachers of Family Medicine, the Association of Departments of Family Medicine, Family Medicine for America's Health, and the American Academy of Family Physicians (AAFP). We conducted structured phone interviews with these stakeholders (N = 17) to gather input on all topics and on specific survey items. We also requested feedback from members of AFMRD at their annual conference, and from faculty development fellows from the University of Washington Family Medicine Residency Network. Concurrently, we mapped the draft survey items to several current efforts relevant to family medicine training, including the Family Medicine Milestone Project9 and the Council of Academic Family Medicine's Consensus Statement for Procedural Training in Family Medicine Residency.19 The mapping process was reviewed by the entire survey development team.
Pilot Testing
After revising the draft survey to incorporate feedback from stakeholders and address gaps observed from the mapping process, we conducted in-depth cognitive interviews with 4 recent graduates from different programs. After additional revisions, we sent the first pilot survey by e-mail to a national convenience sample of recent graduates (Table). The results of this pilot test showed areas for improvement and a need to shorten the survey. To test questions for their impact on survey length, 2 different versions of the survey were created for the second pilot test.
To ensure that the pilot tests were representative of the body of family physicians in the United States, we compared responses to aggregate data from the ABFM's 2013–2014 recertification and certification examination registration questionnaire, the AAFP's publically available member census data,20 and recent published studies about US family physicians. Our pilot participants were more likely to provide maternity and pediatric care,1,2 and less likely to primarily practice urgent or hospital care21 or be in solo practice arrangements.22 This is likely due to our sample being younger and having fewer years in practice.
After completing the second pilot and reviewing the results in detail, the longer version of the survey from the second pilot, which took an average of 12 minutes to complete, was modified and shared with content advisers, the ABFM, and the AFMRD. The final survey had 4 content areas: Practice and Work Schedule, Adequacy of Training and Scope of Practice, Satisfaction, and Professional Activity (box). A description of the resulting variable areas for the main content areas and the final survey are provided as online supplemental material.
Providing direct patient care
Practicing outpatient continuity of care
Principal professional activity
Principal practice site type, ownership, size, and specialty mix
Other practice sites
Number of patient encounters at practice sites
Hours worked in direct patient care, administrative activities, and other areas
Weekend, evening, and on-call care
Content areas (eg, pediatric outpatient care, intensive care, behavioral health care, etc)
Women's health (eg, endometrial biopsy, colposcopy, uterine aspiration, etc)
Orthopedics/musculoskeletal medicine (casting, joint aspiration and injection, musculoskeletal ultrasound)
Genitourinary (vasectomy, neonatal circumcision)
Additional procedures/content areas (eg, cardiac stress test, osteopathic manipulative treatment, management of HIV/AIDS, etc)
Practicing adult inpatient medicine
Practicing obstetrics
Choice of medicine as a profession
Choice of family medicine as a specialty
Residency training
Principal practice: location, partners, employer, hours, income, overall
Burnout and callousness
Certification by other boards
Teaching trainees and other health professionals
Faculty roles
Participation in research
Participation in loan repayment and service obligation programs
Principal practice address
Income
Abbreviation: HIV/AIDS, human immunodeficiency virus infection and acquired immune deficiency syndrome.
Future Uses and Applicability to Other Specialties
Diverse stakeholders in family medicine collaborated on a rigorous methodology to collect data that would meet the ACGME's requirement for surveying graduates and improve residency training, the specialty, and ultimately the health of the public. The resulting content of the survey was different from that of other national public or proprietary surveys in its focus on residency training outcomes and the practice patterns of recent residency graduates.
While residency-level reports were the primary reason for conducting the survey, the survey stakeholder group identified other uses for the data.18 The ABFM deployed the survey in January 2016; Diplomates who graduated from residency in 2013 were alerted via e-mail multiple times throughout the year to complete the survey, which was available in their online portfolio. The inaugural survey had a response rate of 67% (2069 of 3088), and the residency-specific reports were released in March 2017. National-level reports will be shared with other family medicine organizations and the ACGME. Researchers will be able to request data from the ABFM to conduct educational research. Tying the survey to the maintenance of the certification process is easily applicable to other specialties, and has been proven to be a feasible way to gather meaningful data on a majority of residency graduates.
Although the survey we describe was designed for family medicine (the only specialty that currently requires a graduate survey), we believe that surveying graduates will improve training environments and promote collaboration among certifying boards, educators, and the ACGME.23
We hope that our description of the survey content and the survey development process provides a model that could be adopted by other specialties—both allopathic and osteopathic. Residency education is the final step in producing physicians who are essential to meeting the health care needs of Americans.24 With new models of care and recent changes in the accreditation system, the imperative to “ensure a well-trained primary care workforce”25 has never been more urgent, and the need for high-quality, representative data is significant. The National Family Medicine Graduate Survey will enable quality feedback data on training outcomes for residencies to monitor and improve their programs. Other interested specialties may consider replicating or adapting this approach.
