Implementation and Evaluation of Integrating an Electronic Health Record With the ACGME Case Log System

The Accreditation Council for Graduate Medical Education (ACGME) case log system is utilized as an objective assessment of the procedure volume of residents and is important for the accreditation of residency programs in several procedure-based specialties.¹  Although the ACGME guidelines state that all procedures performed by residents should be recorded in the ACGME case log system, numerous studies across specialties have shown that resident cases are underreported and have low accuracy.^2-8 

When comparing ACGME procedure case logs to institutional automated operative case report systems, one study found that only 72% of resident cases across 3 surgical specialties were captured in the ACGME case log, while another found that postgraduate year 1 general surgery residents underreported a mean of 50 cases each year.^3,6  A study of ophthalmology resident case logging found that more than 20% of logged surgeries had inaccuracies, and only 54% of glaucoma surgeries were logged.⁹  Resident case logs have been found to have low accuracy with regard to Current Procedural Terminology (CPT) codes and surgical role, even though a majority of residents believe that accurately logging cases is important to surgical training.^2,7,9,10 

Responses from resident surveys in numerous specialties characterized the case log reporting process as burdensome and difficult to navigate, and noted that the system could become more streamlined by improving the user interface and reducing manual documentation burden.^2,10,11  Past studies identified multiple factors that resulted in increased administrative time for residents, including the ambiguity in categorizing a surgery into multiple component procedures and determining resident role for a case.^2,5  Other work has proposed automated procedure logs or clinical registries to reduce manual documentation,^12,13  but these approaches have not addressed a key issue—linking a procedure to a trainee and then transmitting the data to the ACGME. The difficulty to determine from the electronic health record (EHR) which cases are performed by a resident surgeon also makes it challenging to assess resident case log initiatives.

A more effective reporting system, integrated with existing institutional clinical information systems, might more accurately reflect resident operative volume for the many specialties that have resident case logging requirements. To determine whether an integrated case logging system improved accuracy and resident satisfaction, we created a system integrated with our EHR to transmit resident-logged cases automatically to the ACGME case log system each night.

This study was conducted at the Johns Hopkins Wilmer Eye Institute, an urban academic program. The new case logging system was introduced in 2018, and there were 15 ophthalmology residents in the program throughout the study. Data on resident case logs between July 2016 and June 2020 were extracted from the EHR to assess the efficacy of this new logging system.

To capture procedures performed by ophthalmology residents at the Johns Hopkins Wilmer Eye Institute, we designed and implemented a secure HIPAA-compliant web-based system that was integrated with our EHR (EpicCare, Epic Systems, Verona, WI). The system development took a single developer a few months, supervised by a departmental information technology (IT) manager. The system was launched by clicking on a link in the EHR. To make documentation more efficient, patient demographics and the name of the resident were automatically imported into the new case log record using the EHR's application programming interface. Residents could then choose the correct procedure, attending, location, and role (eg, assistant, surgeon) from menus built into the log system. Once the log entry was completed by the resident and validated by the system, it was saved in a secure HIPAA-compliant local database. No specialized IT resources were required outside existing institutional IT infrastructure.

Once per day, the local case log system uploads cases for that day to the ACGME system using available application programming interfaces. Cases were accepted if valid or, if the case did not satisfy ACGME requirements, an error was returned to the local system where the case could be edited and resubmitted. Once uploaded, trainees could review their logged cases both in the local system and in the ACGME system. Residents were trained on the use of the new system by in-person demonstrations. Tip sheets were also made available online for reference at any time. Residents were reminded to use the system via email and as part of resident meetings.

Since our institution has a resident continuity clinic that reliably generates surgeries performed by residents, we could attribute cases managed in the resident continuity clinic to resident surgeons and thus selected those cases to serve as the measure of how the case log system was adopted. The resident continuity clinic provides residents the opportunity to care for their own group of patients throughout their training all while supervised by faculty in both the clinic and the operating room. We extracted cataract surgeries from our EHR between July 2016 and June 2020 where there was at least one visit with a resident in the resident continuity clinic before surgery, and where the initial postoperative visit was also with a resident in the EHR (Figure). These characteristics are known to be reliably performed by residents at our institution. We also exported the ACGME case log data for this same time window.

Each resident cataract case found in the EHR was matched to its corresponding case in the ACGME data based on date and attending surgeon. The proportion of extracapsular cataract extractions (CPT codes 66982 and 66984) in the EHR that were reported to the ACGME was compared between the periods before the deployment of the local system (6 months), during the transition (6 months), and after the change (2 years). The transition period was included to account for the time needed to train residents, to promote use of the system, and to allow the senior residents to graduate, since their use of the ACGME site was already a habit, with less incentive to change their workflow late in their training. Residents retained the option of logging cases using the standard ACGME-provided web interface throughout the study.

We assessed the efficacy of the integrated system by comparing the adherence of resident logging of cataract surgeries before and after implementation. To measure utilization, we calculated the percentage of total resident-logged cases that were entered in the integrated system for direct upload to the ACGME (instead of cases manually entered in the ACGME platform). This analysis included all surgical procedures, including but not limited to cataract extractions. We used the same time ranges in the previous analysis to compare reported case numbers before the change (July 1, 2016–December 31, 2017), during the transition (January 1, 2018–June 30, 2018), and after the change (July 1, 2018–June 8, 2020). All analyses relied on proportions so the differences in the time allocated to each phase did not affect the analysis.

To evaluate resident satisfaction with the case log system, surveys were administered to all residents before and after implementation to evaluate the frequency and duration of resident use of the system, as well as satisfaction with its accuracy and value. The authors developed the 6-item multiple-choice question survey (provided as online supplementary data) without testing. Two-sample proportion tests were used to compare responses to resident survey questions pre- and post-implementation of the new integrated system. Given the turnover of residents during the study period, the responses to the pre- and post-surveys were not linked to individuals. All statistical analyses were done with R 4.1.2 (The R Foundation).

This study was approved by the Johns Hopkins Medicine Institutional Review Board.

We identified 883 cataract surgeries that could be attributed to residents in the EHR between July 2016 and June 2020. For the time period before the change, there were 409 matches (85%) with the ACGME case log out of 481 resident cataract surgeries identified in the EHR (Table 1). For the time period after the change, there were 300 matches (91%) with the case log out of 329 resident cataract surgeries recorded in the EHR.

We then compared the number of cases entered in our integrated case log system to the total number of surgical cases that were reported to the ACGME (both automated uploads using our system and manual entries using the ACGME web interface). The percentage of total logged cases that were entered using the new integrated system increased from essentially 0% (a few cases were logged during development) to 100% within 2 years (Table 2).

Survey responses from residents before (n=11, 73%) and after (n=14, 93%) the change showed that there was an increase in satisfaction with the new logging system and significant decrease in reported frequency of logging into the ACGME website (Table 3). Before the change, 64% reported logging into the ACGME website at least weekly, compared with 14% after the change (P=.013). The proportion of survey respondents who reported logging into the ACGME website at least monthly dropped from 73% to 29% (P=.032).

Resident reported total time on the procedure log over the course of the academic year did not change with the new system. Resident reported satisfaction was higher after the change, from 27% reporting very satisfied at baseline to 71% after the change (P=.03). Before the change, 55% reported that they were at least moderately satisfied as compared with 100% after the change (P=.006). The proportion of survey respondents who reported being moderately or very dissatisfied with the logging system dropped from 46% to 0% after the change (P=.002). Residents' perceptions of accuracy or value of the logging system did not change.

After an initial transition period when some senior residents preferred to continue with their use of the ACGME case log system, the only steps needed to have residents continue to use the integrated case log system was an annual session with new residents as part of their orientation to this and other information systems. The system was monitored and maintained with effort amounting to less than an hour per week from a departmental IT manager and programmer.

Our results suggest that a resident case log system integrated with an EHR and with automated upload of logged cases to the ACGME more accurately reflects resident operative volume and is preferred by residents over having to access yet another information system to log cases.

This study was able to assess a newly implemented resident case log system integrated with the EHR by analyzing cases from the resident continuity care clinic at our institution. Other studies have approached the challenge of linking a given surgical case or procedure to a resident or fellow using EHRs by adding new distinct data fields that capture trainee involvement. For example, one approach was described in which the metadata needed to link a procedure were added to the documentation completed by cardiology fellows.¹²  This type of approach could be expanded to record trainee involvement in a standardized way, but a concern to be addressed is the inclusion of non-clinical data (eg, the specific role of trainees involved in a case) to clinical notes.

Limitations of our study include the lack of validity evidence for the survey evaluating resident use and satisfaction with the new logging system. Since pretesting was not done on these questions, it is possible that respondents may not have interpreted the questions as intended. Another limitation of this study is the challenge in expanding this type of integrated case log across specialties with varying requirements and institutions with different EHR systems. Despite the generalizability of the system, the details of implementation would have to be customized for each EHR vendor since clinical information platforms differ from institution to institution.

Specific next steps to make a case log system like the one we implemented widely available include: (1) update the ACGME information technology infrastructure to make their programming interface available to many institutions; (2) initiate conversations with common EHR vendors regarding new data elements needed to facilitate case logging; and (3) establish mechanisms by which institutions can share their solutions or collaborate with others on a single solution.

A resident case log system integrated with the institutional EHR increased the number of resident cases recorded, enhanced resident satisfaction, and provided a feasible approach to address the needs of regulatory bodies while reducing trainee administrative burden.

The authors would like to thank Kathy Hackett from the Johns Hopkins Wilmer Eye Institute and Tom Richter from the ACGME information technology group for their contributions to the development of this system, and Kerry Smith from the Johns Hopkins Wilmer Eye Institute for her help extracting EHR data for analysis.