The Resident Readiness Survey: A National Process for Program Directors to Provide Standardized Feedback to Medical Schools About Their Graduates

A standardized Resident Readiness Survey (RRS) process for program directors (PDs) to provide feedback to US MD- and DO-degree granting medical schools about graduates’ preparedness for graduate medical education (GME) was developed and tested nationally. This innovation has the potential to benefit GME programs and medical schools, and ultimately the public at large, because collected data can inform interventions to optimize the undergraduate medical education (UME) to GME transition for all postgraduate year 1 (PGY-1) residents.

Nationally, PDs across a range of specialties have reported relatively low levels of confidence in incoming PGY-1 residents’ readiness for some of the tasks they may assume at the start of residency training.^1–3  However, mechanisms for PDs to provide feedback to schools about their graduates have been fragmented, driven by school-specific initiatives to meet accreditation standards pertaining to evaluation of educational program outcomes.^4,5  PDs have received different surveys at various times from medical schools attended by their PGY-1 residents, and schools have been unable to “benchmark” information about their graduates collected via their school-specific surveys against national data to inform UME curricular quality improvement (QI). Furthermore, PDs may have different specialty-specific expectations of their incoming PGY-1 residents.^6,7  However, specialty-specific data about PGY-1 residents’ readiness for the transition to residency, which could inform specialty organizations’ efforts to optimize graduates’ preparedness, have not been available.^8,9  The Coalition for Physician Accountability (CoPA) UME-GME Review Committee (UGRC) recognized this gap, recommending that, “Early and ongoing specialty-specific resident assessment data should be automatically fed back to medical schools through a standardized process to enhance accountability and to inform continuous improvement of UME programs and learner handovers.”¹⁰  The objective of our innovation was to develop and test the feasibility of a national process for PDs to provide standardized feedback to medical schools about the readiness of their graduates for PGY-1 training.

A standardized RRS instrument “version 1.0” (provided as online supplementary data) was developed through an iterative process that broadly engaged the UME and GME communities (see online supplementary data for details). A national process for RRS administration, data collection, and provision of collected data back to medical schools on an identified basis was developed and tested by the Association of American Medical Colleges (AAMC) in 2 pilot years, in cooperation with the American Association of Colleges of Osteopathic Medicine. Pilot Year (Y) 1 of the RRS process is described as follows.

In the 2020 AAMC Electronic Residency Application Service (ERAS) cycle, updated applicant information included notification that, for medical schools to improve the UME to GME transition, assessment information would be collected from PDs about their GME residents; ERAS applicants were able to opt out of this process.

In February 2020, voluntary participation in the RRS process was opened to all US MD-degree- and DO-degree-granting medical schools with 2019-2020 graduating classes via invitations to the senior medical education and student affairs leaders at each school. Schools that chose to participate signed agreements with the AAMC regarding data confidentiality and RRS data use limitations and agreed not to send school-specific surveys to PDs about their graduates in PGY-1 positions in 2020-2021.¹¹  In November 2020, participating schools’ 2019-2020 graduates in PGY-1 training and their PDs were identified in the AAMC GME Track (a secure resident database and tracking system to assist GME administrators and PDs in GME data collection and management¹² )_, which is populated by data submitted voluntarily by Accreditation Council for Graduate Medical Education (ACGME)-accredited programs about their programs and residents through the National GME Census.¹³  Then (also in November 2020), PDs with any eligible PGY-1 residents (“eligible” PGY-1 residents included participating schools’ 2019-2020 graduates with GME Track and ERAS acknowledgement records) were invited via email to complete, within GME Track, one RRS for each eligible PGY-1 resident, with periodic reminders about outstanding RRSs sent through March 2021. The AAMC then provided school-specific reports of identified RRS data to MD-degree-granting school designees via the AAMC Medical School Profile System portal and to DO-degree-granting school designees via secure email. Y1 process feedback was obtained from participating schools via an evaluation questionnaire in fall 2021 (online supplementary data). An aggregate summary of Y1 RRS process outcomes and data, stratified by specialty, was also released.¹⁴  A similar process was followed for Y2-participating schools’ 2020-2021 graduates with an aggregate Y2 summary released.¹⁵ 

The outcomes measured included program-level outcomes (medical school participation and PD response rates); RRS completion data (eg, completion rates, average completion time); PGY-1 residents’ coverage outcomes (percentages of eligible PGY-1 residents among all participating schools’ graduates and percentages of “covered” PGY-1 residents [those for whom RRSs were completed] among all eligible PGY-1 residents); and participating schools’ feedback about the RRS process and utility of RRS data.

The RRS dataset, de-identified for analysis, included individual-level data for all eligible PGY-1 residents and for all PDs with at least one eligible PGY-1 resident in their programs. Tests on the equality of proportions were used to determine significant differences between proportions, and analysis of variance tested the significance of differences between means, reporting 2-sided P values and 95% confidence intervals (CIs).

The AAMC Human Subjects Office determined this study to be exempt from further Institutional Review Board review, as defined in 45 CFR 46, since this study uses de-identified data.

As shown in Table 1, the school participation rate increased from Y1 (77 of 176, 43.8%) to Y2 (131 of 181, 72.4%) by 28.6 percentage points (95% CI 18.8-38.4; P<.001). Similar percentages of invited PDs responded in Y1 (1786 of 2847, 62.7%) and Y2 (2107 of 3406, 61.9%; P=.48). Across both years combined, PD response rates varied by the number of eligible PGY-1 residents in their programs (from 58.1% [883 of 1520] for PDs with 1 eligible resident to 65.5% [413 of 631] for PDs with 10 or more eligible residents; P=.001) and by specialty (P<.001), from 42.2% (65 of 154, neurology) to 80.1% (109 of 136, internal medicine-pediatrics).

Responding PDs’ RRS completion rates (ie, % of all RRSs sent to the PD that were completed) were high in both years (mean % [SD]: Y1=92.0 [23.1]; Y2=90.6 [24.6]; P=.07; online supplementary data). Among PDs receiving 10 or more RRSs, the mean percentage completed increased by 15 percentage points (95% CI 7.1-22.9), from 66.9% [43%] among 122 PDs in Y1 to 81.9% [34.4%] among 291 PDs in Y2 (P<.001). RRS average completion time was less than 3 minutes per survey in both years.

The percentage of eligible PGY-1 residents among all participating schools’ graduates increased by 2.7 percentage points from Y1 to Y2 (89.0% [10 712 of 12 035] vs 91.7% [18 372 of 20 044]; P<.001; online supplementary data). The percentage of covered PGY-1 residents (PGY-1 residents for whom RRSs were completed out of all eligible PGY-1 residents) was similar in Y1 and Y2 (52.0% [5567 of 10 712] vs 52.7% [9688 of 18 372; P>.05). Across both years, PGY-1 residents’ coverage varied by specialty (P<.001), ranging from 36.4% (155 of 426, neurology) to 68.2% (270 of 396, urology). The per school PGY-1 residents’ coverage percentage was similar in Y1 and Y2 (mean % [SD]: 52.0% [7.9], 77 schools vs 53.5% [7.0], 131 schools; P=.16).

Evaluation questionnaire response rates were similar (P=.38) in Y1 (68 of 77 [88.3%]) and Y2 (109 of 131 [83.2%]). As shown in Table 2, item response distributions in Y1 and Y2 were also similar. Across both years, 10.8% (17 of 158) of respondents indicated that their school’s RRS report had stimulated new curricular QI efforts and 72.8% (115 of 158) indicated that it had supported ongoing curricular QI efforts. Among schools that had previously sent local surveys to PDs, 62.1% (87 of 140) reported PD response rates of 50% or less. Shown in the Box are examples of initial uses of RRS data reported by participating schools.

The feasibility of implementing a national process for PDs to provide standardized feedback to medical schools about the preparedness of their graduates for PGY-1 training was demonstrated over 2 years. Overall PD response rates and RRS completion rates suggest that the process worked well for many PDs; the process was also positively evaluated by participating medical schools.

Overall RRS PD response rates compared favorably with others’ experiences. Cooper et al reported a 20% response rate to a survey sent to PDs in 4 specialties, and the National Resident Matching Program reported less than 35% response rates for surveys sent to PDs in 2021 to 2022 about virtual recruitment experiences.^16,17  Recently reported response rates for specialty-specific PD surveys ranged from 42% for general surgery¹⁸  to 57% for internal medicine.⁶ 

Among responding PDs, differences we observed in PD response rates and RRS completion rates by the number of RRSs received suggest that PDs of larger programs may particularly appreciate the value of the RRS process but may also face challenges in completing the large numbers of RRSs received. The significant increase in the RRS completion rate in Y2 compared to Y1 among responding PDs who received 10 or more RRSs is notable in this context. Ongoing PD community engagement to incorporate their feedback for process improvements, demonstrate the utility of information provided by PDs to medical schools, and explore the value of national benchmarking data to specialty organizations working to improve the transition to residency may be key to sustaining and potentially even increasing PD response rates and RRS completion rates.

Importantly, the RRS is only one way of sharing information about the transition to residency. For a truly productive 2-way UME-GME system of communication, expansion of information shared “up” from UME to GME—beyond the Medical School Performance Evaluation—is also needed. One approach, aligned with another recommendation by the CoPA UGRC,¹⁰  is the educational handover. This post-Match UME-to-GME communication, about the incoming PGY-1 resident’s current strengths and areas for improvement (also referred to as a “warm handover”), could inform individualized learning plans at the start of GME^19,20  and warrants similar consideration of a national approach.

The RRS, as a standardized survey about all PGY-1 residents, may include some items that PDs do not consider particularly relevant to their specialty and may not include others that PDs would consider highly relevant to their specialty. Periodic review and refinement of the RRS, with input from PDs across specialties to ensure the items are meaningful and maintain relevance across specialties, will be important. Furthermore, the PD perspective has not necessarily been routinely incorporated into UME curriculum revision processes²¹ ; the RRS can provide a means to collect PD feedback on UME curricular strengths and areas for improvement to inform both school-level and national curriculum QI efforts.

Although international medical school graduates (IMGs) currently comprise about 23% of residents in US ACGME-accredited GME programs,²²  participation was not open to international medical schools due to resource limitations. Establishment of an analogous RRS process for IMGs could further improve the preparedness of all PGY-1 residents entering ACGME-accredited GME programs.

Finally, validity evidence for the RRS must be considered. Of 5 sources of validity evidence, initially described by Messick and since widely adopted,²³  there is some content validity evidence for the RRS (as further described in the Resident Readiness Survey Instrument Development document provided as online supplementary data, and through the collection of evaluation data from participating schools regarding RRS content and utility), but sources of evidence regarding response processes, internal structure, relations to other variables, and consequences of the RRS remain to be fully examined. Based on the results of the Pilot Y1 and Y2, the AAMC will continue to administer the RRS on an annual basis.

It is feasible to administer a standardized survey to PDs nationally about the readiness of their PGY-1 residents. School participation substantially increased from Y1 to Y2, with most of the invited schools participating in Y2. Response rates among invited PDs were generally high, as were RRS completion rates among responding PDs.

The authors would like to thank the following individuals, all full-time employees of the Association of American Medical Colleges (AAMC), who served on the AAMC’s Resident Readiness Survey Workgroup: Marie Caulfield (retired), Manager, Data Operations and Services; Virginia Bush, Project Management Specialist, Academic Affairs; Keith Horvath, Senior Director, Health Care Affairs; Mallory Lee, Business Operations Specialist, Academic Affairs; Cecilia Barry, Senior Corporate Counsel, Legal Services; Lynn Shaull, Senior Research Analyst, Academic Affairs, Amy Addams, Director, Student Affairs Strategy and Alignment, Academic Affairs and Erin Helbling, Senior Specialist, Services. They also thank Dolores Mullikin, Tripler Army Medical Center Clerkship Director and Uniformed Services University health professions education doctoral candidate, for her service on the workgroup, and Whitney Staiger, Business Operations Specialist, AAMC for her assistance.

The online version of this article contains the AAMC Resident Readiness Survey, survey instrument development, task force roster, AAMC Resident Readiness Pilot Evaluation Survey, Resident Readiness Survey completion data, and postgraduate year 1 residents’ coverage outcomes.

Funding: The authors report no external funding source for this study.

Conflict of interest: Lisa Howley, PhD, Douglas Grbic, PhD, Amy Jayas, MPH, Lindsay B. Roskovensky, BA, and Dorothy A. Andriole, MD, are full-time employees of the Association of American Medical Colleges (AAMC) and also served on the AAMC’s internal Resident Readiness Survey Workgroup. Mark R. Speicher, PhD, MHA, is a full-time employee of the American Association of Colleges of Osteopathic Medicine, whose member Colleges of Osteopathic Medicine participate in the Resident Readiness Survey process.

An abstract of this work was presented at the Association of American Medical Colleges’ Learn Serve Lead meeting, November 11-15, 2022, Nashville, TN, and published; Grbic D, Andriole DA, Mullikin D, Howley LD. Program directors’ assessment of postgraduate year 1 residents’ readiness for graduate medical education: lessons and insights from the Association of American Medical Colleges inaugural 2020-2021 resident readiness survey. Acad Med. 2022;97(suppl 11):128. doi:10.1097/ACM.0000000000004888