ABSTRACT
As resident “index” procedures change in volume due to advances in technology or reliance on simulation, it may be difficult to ensure trainees meet case requirements. Training programs are in need of metrics to determine how many residents their institutional volume can support.
As a case study of how such metrics can be applied, we evaluated a case distribution simulation model to examine program-level mediastinoscopy and endobronchial ultrasound (EBUS) volumes needed to train thoracic surgery residents.
A computer model was created to simulate case distribution based on annual case volume, number of trainees, and rotation length. Single institutional case volume data (2011–2013) were applied, and 10 000 simulation years were run to predict the likelihood (95% confidence interval) of all residents (4 trainees) achieving board requirements for operative volume during a 2-year program.
The mean annual mediastinoscopy volume was 43. In a simulation of pre-2012 board requirements (thoracic pathway, 25; cardiac pathway, 10), there was a 6% probability of all 4 residents meeting requirements. Under post-2012 requirements (thoracic, 15; cardiac, 10), however, the likelihood increased to 88%. When EBUS volume (mean 19 cases per year) was concurrently evaluated in the post-2012 era (thoracic, 10; cardiac, 0), the likelihood of all 4 residents meeting case requirements was only 23%.
This model provides a metric to predict the probability of residents meeting case requirements in an era of changing volume by accounting for unpredictable and inequitable case distribution. It could be applied across operations, procedures, or disease diagnoses and may be particularly useful in developing resident curricula and schedules.
What was known and gap
Residency programs need to assess whether institutional case volumes are sufficient to support resident-level volume requirements.
What is new
A computer model to simulate case distribution, and to predict the likelihood of all residents in a given program to achieve required case volumes.
Limitations
Single institution study reduces generalizability; historical data may not reflect future trends in procedural volume.
Bottom line
The model is applicable across a range of procedures and may be useful in developing resident schedules.
Introduction
The current surgical training environment is an evolving landscape, with new technology transforming techniques and educational curricula, including increased use of simulation.1,2 Examples include the introduction of ultrasound, which has dramatically changed the techniques for central line placement and thoracentesis.3,4 Many hysterectomies have shifted from open approaches to laparoscopic, despite unchanged requirements for open hysterectomies.5,6 Similarly, in thoracic surgery, the biopsy of mediastinal lymph nodes by mediastinoscopy has decreased, as endobronchial ultrasound (EBUS)–guided biopsies have shown similar efficacy.7 Programs must account for new procedures, and for the distribution of “index” cases or procedures required for board certification in most surgical specialties. Thus, there is a critical need for metrics that allow program leaders to assess and ensure that all residents receive equitable experience in key procedural or disease-related requirements, while accounting for the many factors that influence trainees' experience (eg, inequitable case distribution).
When program leaders consider an adequate experience for trainees, the procedural volume necessary to train residents is not simply equal to the number of trainees multiplied by the number of cases/procedures required (ie, 4 trainees × 10 cases required per trainee ≠ 40 total cases at the institution). Rather, the case volume required is higher than expected, due to the need to consider multiple additional factors, such as schedules, trainee availability, and duty hour limits that influence a given trainee's experience.8 A more rigorous approach incorporating these variables can be accomplished using mathematical modeling, which has demonstrated the ability to predict whether a program can meet the case requirements of its residents based on program size and case volume.8
We examined the utility of simulation in modeling the ability of a training program to meet case requirements, utilizing an emerging procedure in a thoracic surgery—EBUS—which has led to decreased mediastinoscopy numbers. We applied the case distribution simulation model to assess the total program volume required to train thoracic surgery residents for these procedures.
Methods
Data Source
We obtained mediastinoscopy and EBUS procedural volumes from a retrospective review of 2011–2013 billing records of a single academic tertiary care center with a high-volume thoracic surgery service to obtain data on all possible training/learning opportunities.
The study was deemed exempt and approved by our Institutional Review Board.
Analysis
We used historical case volumes for EBUS and mediastinoscopy to determine the mean number of cases performed annually. A mathematical model was built to predict the number of case events within a defined period of time. The methods behind the model have been published previously,8 and a more detailed explanation is also supplied (provided as online supplemental material). The model uses information related to the institution's case volume and case interarrival time (time between case occurrences) to create a distribution (eg, normal, Poisson) of interarrival times. This distribution can be used to predict future case arrivals at the specific institution to account for when cases may realistically occur. This is used to simulate a trainee's operative experience within a certain work or on-call schedule.
We used mathematical modeling, similar to Monte Carlo simulation, to predict the occurrence and distribution of future case arrivals. Assignment to the trainee was based on input of a hypothetical trainee schedule compliant with the Accreditation Council for Graduate Medical Education duty hour limits. The model continuously determined case interarrival time, and assigned cases to trainees until stopping criteria were reached. We used 2 years as the stopping criterion to reflect the amount of time required to complete a traditional thoracic surgery residency/fellowship at our institution. Results were aggregated for each procedure once the specified stopping criterion was reached, and we simulated 10 000 two-year periods of residency training. The model then determined the probability of all trainees meeting case requirements according to the guidelines for their respective pathways.
We set the case requirement for each training pathway according to the American Board of Thoracic Surgery (ABTS) operative requirements,9 using 2 different scenarios: (1) mediastinoscopy requirements for residents training prior to July 1, 2012 (pre-2012: general thoracic pathway, 25; cardiothoracic pathway, 10), and (2) those implemented following July 1, 2012 (post-2012: general thoracic pathway, 15; cardiothoracic pathway, 10; EBUS general thoracic pathway, 10; EBUS cardiothoracic pathway, no specific requirement; figure 1a–d). Prior to 2012, EBUS case requirements did not exist, and this scenario was not analyzed. The model was run for each pre-2012 and post-2012 scenario to evaluate the effect of various case requirements on the total procedural volume necessary to train all residents (4 trainees; 2 fellows per year). Our primary outcome was the probability that all residents in the program would achieve the minimum required number of a given case within the defined period.
Simulator Interface
Note: (a) Simulator interface displaying the individual procedure and corresponding case volume entry. (b) Simulator interface with entry of individual trainee case requirements by procedure. (c) Specification of trainees' schedule to designate case assignment in the simulator with designation of the maximum number of procedures per day a trainee can perform. (d) Simulator interface displaying stopping criteria according to the number of days elapsed (2 years/730 days) and designation of the number of repetitive simulations (10 000).
Simulator Interface
Note: (a) Simulator interface displaying the individual procedure and corresponding case volume entry. (b) Simulator interface with entry of individual trainee case requirements by procedure. (c) Specification of trainees' schedule to designate case assignment in the simulator with designation of the maximum number of procedures per day a trainee can perform. (d) Simulator interface displaying stopping criteria according to the number of days elapsed (2 years/730 days) and designation of the number of repetitive simulations (10 000).
Results
Single Year Simulation
We observed 43 ± 6.2 (mean ± SD) mediastinoscopies and 19 ± 1.1 endobronchial ultrasound procedures performed per year using the institution's historical case volumes (2011–2013). The simulator was run using these mean values with case requirements specified to reflect the post-2012 era. When simulating a single year, there was considerable variability in case distribution among trainees (figure 2a for EBUS and figure 2b for mediastinoscopy). As institutional volume may vary from year to year, for example, a single year with low volume could result in 1 or more trainees not achieving adequate numbers. This was simulated using a mean annual case volume of 35 mediastinoscopies (figure 2c). Despite adequate overall institutional volume, the inequities in procedure distribution among trainees resulted in more than enough mediastinoscopies for 3 trainees (Card-A, Card-B, and Thor-B) and insufficient cases for a fourth (Thor-A; figure 2c).
Simulator Output
Note: Simulator output displaying a single simulated year, showing the number of endobronchial ultrasound (a) and the number of mediastinoscopies (b) performed by 4 trainees, in which all trainees achieve their minimum case volume. An alternative simulated year (c) shows the inability of 1 of 4 residents (Thor-A) to meet required operative volume. Gray represents cases completed toward American Board of Thoracic Surgery (ABTS) requirement, white represents cases beyond ABTS requirement, and black represents cases deficient of ABTS requirement.
Abbreviations: Card-A, Card-B, cardiothoracic pathway trainees A and B, respectively; Thor-A, Thor-B, thoracic pathway trainees A and B, respectively.
Simulator Output
Note: Simulator output displaying a single simulated year, showing the number of endobronchial ultrasound (a) and the number of mediastinoscopies (b) performed by 4 trainees, in which all trainees achieve their minimum case volume. An alternative simulated year (c) shows the inability of 1 of 4 residents (Thor-A) to meet required operative volume. Gray represents cases completed toward American Board of Thoracic Surgery (ABTS) requirement, white represents cases beyond ABTS requirement, and black represents cases deficient of ABTS requirement.
Abbreviations: Card-A, Card-B, cardiothoracic pathway trainees A and B, respectively; Thor-A, Thor-B, thoracic pathway trainees A and B, respectively.
Multiple Year Simulation and Trainee Certification Rates
By changing inputs in the simulation, we were able to assess the probability of individual residents, or all trainees within a program, achieving required case volumes by running multiple year simulations (10 000). We analyzed mediastinoscopy case requirements by time period (pre-2012 versus post-2012) and found that all 4 trainees would achieve requirements in 88% of occurrences under current post-2012 standards (figure 3a). However, prior to 2012, this would have occurred only 6% of the time, due to the higher required number of mediastinoscopies (figure 3b). If our institution's mediastinoscopy volume is reduced by 20% (mean of 43 to a mean of 35 annual cases), our ability to train all 4 residents, even in the post-2012 era of reduced requirements, would decrease from 88% to 55% (figure 3c). For EBUS cases, using a mean of 19 annual cases per year, the probability that all 4 trainees would meet case requirements was reduced to 23% (figure 3d).
Variable Likelihood of Fellows Meeting Requirements
Note: (a) Simulator interface displaying results indicating that all 4 trainees would achieve mediastinoscopy requirements in 88% of occurrences under current standards, assuming institutional case volume of 43 cases per year. (b) However, when using pre-2012 requirements, all 4 trainees would meet requirements in only 6% of occurrences. (c) If current mediastinoscopy volume was reduced by 20% (mean of 43 to a mean of 35 annual cases), the ability to train all 4 residents under post-2012 requirements would decrease from 88% to 55%. (d) For endobronchial ultrasound cases, using a mean of 19 annual cases per year, the probability that all 4 trainees would meet case requirements was 23%.
Variable Likelihood of Fellows Meeting Requirements
Note: (a) Simulator interface displaying results indicating that all 4 trainees would achieve mediastinoscopy requirements in 88% of occurrences under current standards, assuming institutional case volume of 43 cases per year. (b) However, when using pre-2012 requirements, all 4 trainees would meet requirements in only 6% of occurrences. (c) If current mediastinoscopy volume was reduced by 20% (mean of 43 to a mean of 35 annual cases), the ability to train all 4 residents under post-2012 requirements would decrease from 88% to 55%. (d) For endobronchial ultrasound cases, using a mean of 19 annual cases per year, the probability that all 4 trainees would meet case requirements was 23%.
Program Volume and Trainee Certification Rates
Program volume demonstrates a nonlinear relationship to the probability of all program trainees achieving case certification. We analyzed institutional EBUS volume and the probability of all 4 residents achieving minimum case numbers using post-2012 ABTS case requirements (figure 4a). Based on this, our institution would need an average of 35 EBUS cases per year to ensure a 95% probability that all trainees meet minimum cases numbers.
Relationship of Program Certification Rates to Case Volume
Note: Relationship between institutional endobronchial ultrasound volume (a) or mediastinoscopy volume (b) and the probability of all trainees meeting board case requirements (post-2012). (c) Effect of the change in American Board of Thoracic Surgery case requirements (pre-2012: general thoracic pathway, 25; cardiothoracic pathway, 10; versus post-2012: general thoracic pathway, 15; cardiothoracic pathway, 10) on the institutional volume of mediastinoscopy cases required to train all residents.
Relationship of Program Certification Rates to Case Volume
Note: Relationship between institutional endobronchial ultrasound volume (a) or mediastinoscopy volume (b) and the probability of all trainees meeting board case requirements (post-2012). (c) Effect of the change in American Board of Thoracic Surgery case requirements (pre-2012: general thoracic pathway, 25; cardiothoracic pathway, 10; versus post-2012: general thoracic pathway, 15; cardiothoracic pathway, 10) on the institutional volume of mediastinoscopy cases required to train all residents.
Similarly, when evaluating mediastinoscopy volume utilizing post-2012 requirements, a similar nonlinear relationship between program case volume and trainee certification rates is seen (figure 4b). As shown before, our annual mean of 43 mediastinoscopies indicates that all trainees would complete case requirements 88% of the time. To achieve a 95% probability, our institutional case volume would need to increase 12% to an average of 48 mediastinoscopies per year. We evaluated how the pre-2012 and post-2012 era requirements changed the program mediastinoscopy volume needed (figure 4c). This showed that, under the pre-2012 case requirements, the number of cases needed to train 4 residents with a probability of 95% was approximately 72 cases per year. Lowering the requirements (the post-2012 requirements) shifted the curve to the left, resulting in only 48 cases needed per year.
Discussion
Our results demonstrate the utility of mathematical modeling to evaluate the ability of a residency program to achieve case requirements for its trainee complement using current information (eg, training schedules, case volume). Our findings demonstrate that while the number of cases at an institution may be perceived as adequate, the necessary case volume to train a given number of residents may in fact be much greater due to inequities in case distribution. Anticipating the number of cases needed to support trainees is more sophisticated than simply multiplying the number of residents by the required number of cases per trainee. We found that when accounting for the many factors influencing trainees' experience, higher annual volumes of mediastinoscopies are actually necessary, as the current volume is not adequate to train residents without “poaching,” or trading cases when residents are off service. A working simulator with instructions on how to enter data is available for any group to use at http://cheps.engin.umich.edu/tools/stereo.
Previously, simulation has been utilized to demonstrate the minimum number of institutional cases required to provide trainees with a sufficient heart and lung transplantation operative experience.8 We build on this work by demonstrating its utility for 2 elective procedures, while accounting for the varying case requirements for residents based on their training pathways. This approach may more accurately reflect real-life training scenarios, and is an important consideration for program directors when evaluating the minimum case volume necessary to support a given residency complement. We also demonstrated the substantial impact that changing technology can have on the institutional volume needed to meet volume benchmarks for all trainees.
This study has several limitations. The annual procedural volume and distribution of procedures were based on data from a single institution, and we used historical institutional data to predict future procedure arrivals, which may not reflect future procedural volume. Finally, this simulation model does not account for all variables that may influence a resident's ability to participate in a specific number of cases, such as last minute schedule changes or residents “swapping” cases.
Conclusion
Our study demonstrates the utility of a simulation model that program directors can use to anticipate and demonstrate a program's capacity to accommodate a specific number of residents, using existing institutional procedure or case volume data.
References
Author notes
Funding: Dr. Grenda was supported by the Agency for Healthcare Research and Quality grant 2T32HS000053. Dr. Ballard was supported by a research fellowship grant from The Plastic Surgery Foundation.
Competing Interests
Conflict of interest: The authors declare they have no competing interests.
Editor's Note: The online version of this article contains a detailed description of the mechanics of the simulator model.