Computer Modeling to Evaluate the Impact of Technology Changes on Resident Procedural Volume

The current surgical training environment is an evolving landscape, with new technology transforming techniques and educational curricula, including increased use of simulation.^1,2  Examples include the introduction of ultrasound, which has dramatically changed the techniques for central line placement and thoracentesis.^3,4  Many hysterectomies have shifted from open approaches to laparoscopic, despite unchanged requirements for open hysterectomies.^5,6  Similarly, in thoracic surgery, the biopsy of mediastinal lymph nodes by mediastinoscopy has decreased, as endobronchial ultrasound (EBUS)–guided biopsies have shown similar efficacy.⁷  Programs must account for new procedures, and for the distribution of “index” cases or procedures required for board certification in most surgical specialties. Thus, there is a critical need for metrics that allow program leaders to assess and ensure that all residents receive equitable experience in key procedural or disease-related requirements, while accounting for the many factors that influence trainees' experience (eg, inequitable case distribution).

When program leaders consider an adequate experience for trainees, the procedural volume necessary to train residents is not simply equal to the number of trainees multiplied by the number of cases/procedures required (ie, 4 trainees × 10 cases required per trainee ≠ 40 total cases at the institution). Rather, the case volume required is higher than expected, due to the need to consider multiple additional factors, such as schedules, trainee availability, and duty hour limits that influence a given trainee's experience.⁸  A more rigorous approach incorporating these variables can be accomplished using mathematical modeling, which has demonstrated the ability to predict whether a program can meet the case requirements of its residents based on program size and case volume.⁸ 

We examined the utility of simulation in modeling the ability of a training program to meet case requirements, utilizing an emerging procedure in a thoracic surgery—EBUS—which has led to decreased mediastinoscopy numbers. We applied the case distribution simulation model to assess the total program volume required to train thoracic surgery residents for these procedures.

We obtained mediastinoscopy and EBUS procedural volumes from a retrospective review of 2011–2013 billing records of a single academic tertiary care center with a high-volume thoracic surgery service to obtain data on all possible training/learning opportunities.

The study was deemed exempt and approved by our Institutional Review Board.

We used historical case volumes for EBUS and mediastinoscopy to determine the mean number of cases performed annually. A mathematical model was built to predict the number of case events within a defined period of time. The methods behind the model have been published previously,⁸  and a more detailed explanation is also supplied (provided as online supplemental material). The model uses information related to the institution's case volume and case interarrival time (time between case occurrences) to create a distribution (eg, normal, Poisson) of interarrival times. This distribution can be used to predict future case arrivals at the specific institution to account for when cases may realistically occur. This is used to simulate a trainee's operative experience within a certain work or on-call schedule.

We used mathematical modeling, similar to Monte Carlo simulation, to predict the occurrence and distribution of future case arrivals. Assignment to the trainee was based on input of a hypothetical trainee schedule compliant with the Accreditation Council for Graduate Medical Education duty hour limits. The model continuously determined case interarrival time, and assigned cases to trainees until stopping criteria were reached. We used 2 years as the stopping criterion to reflect the amount of time required to complete a traditional thoracic surgery residency/fellowship at our institution. Results were aggregated for each procedure once the specified stopping criterion was reached, and we simulated 10 000 two-year periods of residency training. The model then determined the probability of all trainees meeting case requirements according to the guidelines for their respective pathways.

We set the case requirement for each training pathway according to the American Board of Thoracic Surgery (ABTS) operative requirements,⁹  using 2 different scenarios: (1) mediastinoscopy requirements for residents training prior to July 1, 2012 (pre-2012: general thoracic pathway, 25; cardiothoracic pathway, 10), and (2) those implemented following July 1, 2012 (post-2012: general thoracic pathway, 15; cardiothoracic pathway, 10; EBUS general thoracic pathway, 10; EBUS cardiothoracic pathway, no specific requirement; figure 1a–d). Prior to 2012, EBUS case requirements did not exist, and this scenario was not analyzed. The model was run for each pre-2012 and post-2012 scenario to evaluate the effect of various case requirements on the total procedural volume necessary to train all residents (4 trainees; 2 fellows per year). Our primary outcome was the probability that all residents in the program would achieve the minimum required number of a given case within the defined period.

We observed 43 ± 6.2 (mean ± SD) mediastinoscopies and 19 ± 1.1 endobronchial ultrasound procedures performed per year using the institution's historical case volumes (2011–2013). The simulator was run using these mean values with case requirements specified to reflect the post-2012 era. When simulating a single year, there was considerable variability in case distribution among trainees (figure 2a for EBUS and figure 2b for mediastinoscopy). As institutional volume may vary from year to year, for example, a single year with low volume could result in 1 or more trainees not achieving adequate numbers. This was simulated using a mean annual case volume of 35 mediastinoscopies (figure 2c). Despite adequate overall institutional volume, the inequities in procedure distribution among trainees resulted in more than enough mediastinoscopies for 3 trainees (Card-A, Card-B, and Thor-B) and insufficient cases for a fourth (Thor-A; figure 2c).

By changing inputs in the simulation, we were able to assess the probability of individual residents, or all trainees within a program, achieving required case volumes by running multiple year simulations (10 000). We analyzed mediastinoscopy case requirements by time period (pre-2012 versus post-2012) and found that all 4 trainees would achieve requirements in 88% of occurrences under current post-2012 standards (figure 3a). However, prior to 2012, this would have occurred only 6% of the time, due to the higher required number of mediastinoscopies (figure 3b). If our institution's mediastinoscopy volume is reduced by 20% (mean of 43 to a mean of 35 annual cases), our ability to train all 4 residents, even in the post-2012 era of reduced requirements, would decrease from 88% to 55% (figure 3c). For EBUS cases, using a mean of 19 annual cases per year, the probability that all 4 trainees would meet case requirements was reduced to 23% (figure 3d).

Program volume demonstrates a nonlinear relationship to the probability of all program trainees achieving case certification. We analyzed institutional EBUS volume and the probability of all 4 residents achieving minimum case numbers using post-2012 ABTS case requirements (figure 4a). Based on this, our institution would need an average of 35 EBUS cases per year to ensure a 95% probability that all trainees meet minimum cases numbers.

Similarly, when evaluating mediastinoscopy volume utilizing post-2012 requirements, a similar nonlinear relationship between program case volume and trainee certification rates is seen (figure 4b). As shown before, our annual mean of 43 mediastinoscopies indicates that all trainees would complete case requirements 88% of the time. To achieve a 95% probability, our institutional case volume would need to increase 12% to an average of 48 mediastinoscopies per year. We evaluated how the pre-2012 and post-2012 era requirements changed the program mediastinoscopy volume needed (figure 4c). This showed that, under the pre-2012 case requirements, the number of cases needed to train 4 residents with a probability of 95% was approximately 72 cases per year. Lowering the requirements (the post-2012 requirements) shifted the curve to the left, resulting in only 48 cases needed per year.

Our results demonstrate the utility of mathematical modeling to evaluate the ability of a residency program to achieve case requirements for its trainee complement using current information (eg, training schedules, case volume). Our findings demonstrate that while the number of cases at an institution may be perceived as adequate, the necessary case volume to train a given number of residents may in fact be much greater due to inequities in case distribution. Anticipating the number of cases needed to support trainees is more sophisticated than simply multiplying the number of residents by the required number of cases per trainee. We found that when accounting for the many factors influencing trainees' experience, higher annual volumes of mediastinoscopies are actually necessary, as the current volume is not adequate to train residents without “poaching,” or trading cases when residents are off service. A working simulator with instructions on how to enter data is available for any group to use at http://cheps.engin.umich.edu/tools/stereo.

Previously, simulation has been utilized to demonstrate the minimum number of institutional cases required to provide trainees with a sufficient heart and lung transplantation operative experience.⁸  We build on this work by demonstrating its utility for 2 elective procedures, while accounting for the varying case requirements for residents based on their training pathways. This approach may more accurately reflect real-life training scenarios, and is an important consideration for program directors when evaluating the minimum case volume necessary to support a given residency complement. We also demonstrated the substantial impact that changing technology can have on the institutional volume needed to meet volume benchmarks for all trainees.

This study has several limitations. The annual procedural volume and distribution of procedures were based on data from a single institution, and we used historical institutional data to predict future procedure arrivals, which may not reflect future procedural volume. Finally, this simulation model does not account for all variables that may influence a resident's ability to participate in a specific number of cases, such as last minute schedule changes or residents “swapping” cases.

Our study demonstrates the utility of a simulation model that program directors can use to anticipate and demonstrate a program's capacity to accommodate a specific number of residents, using existing institutional procedure or case volume data.