Using a Web-Based Survey for the Annual Internal Medicine Residency Evaluation

The demands on departments administering graduate medical education programs are numerous. Resident scheduling, coordinating conferences, and managing evaluations are especially time consuming, and would benefit from approaches to improve efficiency. We report on efficiency and quality improvement gains achieved by switching annual surveys required by the Residency Review Committee (RRC) from a paper-based to a web-based system.

Our internal medicine residency program used a 15-page paper survey to fulfill the RRC requirement for resident evaluation of the program. The survey asked questions to assess clinic and inpatient rotations and residents' perceptions about duty hours, stress and fatigue, attending evaluations, and how to improve patient care and hospital services. The survey was distributed and collected during our annual resident retreat in May, results and comments aggregated, and discussed at a separate resident meeting later in the summer. Collating this feedback and distributing it to the proper individuals involved handling 1000 pieces of paper and took an estimated 50 administrative hours annually.

In academic year 2009–2010, we changed the survey to a web-based format, and asked residents to complete it during the week prior to our May retreat. We used Surveymonkey.com to transfer to a paperless survey. We chose this web-based service for many reasons. It was relatively inexpensive, costing $20 per month for a professional account. It was easy to use, and provided unlimited questions, up to 1000 responses per month. Other features such as skip logic and downloading of responses were also useful. Editing the paper survey and transferring it to a web-based format took approximately 15 hours.

The benefits of the web-based survey were seen immediately. Of the 76 residents in the program, 63 (83%) logged on and completed the survey before the May resident retreat. By the start of the retreat, the program directors were able to aggregate the resident feedback data, and could incorporate graphic representations of the data into PowerPoint presentations. Processing the data required approximately 90 minutes. Break-out sessions at the retreat were more effective because residents and program administrators would analyze and respond to the resident feedback in real time. We feel the quality of response to resident feedback was enhanced by this quicker reaction to comments and assessments.

Residents indicated that it was easier to respond using a web-based survey because it allowed them to type detailed comments. It took the residents an average of 45 minutes to complete the survey. To maintain anonymity of the respondents, residents were required to send a separate e-mail to the program manager when the survey was completed.

We surveyed the other internal medicine residency training programs in Southern California to determine the type of survey used for the ACGME-required annual program evaluation. A web-based survey was developed for this purpose, and emailed to the 17 programs in July 2010. We made follow-up phone calls to the program managers/coordinators. Nine programs responded to the survey directly, 6 gave limited responses over the phone, and 2 did not respond (overall response rate of 88%).

Of the 15 programs, 12 used a web-based survey system (80%) and 3 used a paper-based system (20%). Among programs that provided information regarding the specific type of web-based survey used, Surveymonkey.com was the most popular, and Verinform was the second most popular. Programs reported they had used web-based surveys from 1 to 7 years. The 3 programs that used paper surveys were smaller (an average of about 26 residents), and 2 were in private, community hospitals.

Programs are looking for more efficient ways to carry out ACGME required program administration and evaluation activities. Our experience in switching to a web-based system was quite favorable, with positive feedback from the residency program directors and administrators, as well as the residents. In fact, our residency program will be using web-based surveys in other applications, for example, evaluating the experience of interviewees during this year's intern applicant interviews.

Web-based surveys are an important and prevalent tool, used by the majority of internal medicine residency programs in Southern California. They are relatively straight-forward to adopt, and can be maintained on commercial servers safely and at low cost. The option for anonymous responses, while separately tracking who responded, ensures frank responses and data confidentiality. We expect other residency programs nationally to show a similar trend in the use of web-based surveys, and we recommend this approach to programs still using paper-based surveys.