Teaching Systems-Based Competency in Anesthesiology Residency: Development of an Education and Assessment Tool Open Access

In 1999, the Accreditation Council for Graduate Medical Education (ACGME) identified 6 competencies for resident education, patient care, medical knowledge, interpersonal and communication skills, professionalism, practice-based learning, and systems-based practice.1 This move toward competency-based education was accompanied by a mandate to assess all residents in each competency; programs have to demonstrate that residents are learning the required material and must develop education and assessments tools to meet these mandates.

A particularly difficult competency to teach and assess is systems-based practice (SBP). Given its broad scope to “demonstrate awareness of and responsiveness to the larger context and systems of health care, and the ability to effectively call on systems resources to provide care of optimal value,” each specialty is charged with developing its own education and assessment plan, specifically tailored to its unique challenges. Hospital-based specialties, including anesthesiology, can have difficulties defining SBP that is relevant to their practice. For example, in a typical anesthesia practice, costs are divided across the hospital, the pharmacy system, and the anesthesia group. In addition, a unique unit-based anesthesia billing system differentiates the specialty from others. These challenges are not systematically discussed, taught, or evaluated in most anesthesiology residencies, and enhancing resident education about the financial implications of practice provides excellent opportunities to target SBP.

The cost of health care is addressed in the ACGME's resident expectations in SBP, which state that residents are expected to “incorporate considerations of cost awareness and risk-benefit analysis in patient and/or population based care as appropriate.” We hypothesized that our residents had a limited understanding of the cost of anesthetic drugs, and that we could improve this knowledge with a relatively simple and easily reproducible tool.

We developed this tool as a partial answer to teaching and assessing SBP. A recent article suggests that SBP in anesthesiology is a group-based competency, and focuses on group attributes in the health care system, rather than on the behavior of the individual.2 As a result, the authors proposed a group-based resident education module. Eiser and Connaughton-Storey3 described a 2-week supervised experience in SBP that included medical residents, representatives of home health care services, hospice care, pharmacy services, laboratory services, and others, and concluded that an integrated multidisciplinary approach is best. Although there are group dynamics inherent in SBP for all specialties, anesthesiologists make clinical decisions individually or in small care teams, and these decisions can have a profound effect when taken cumulatively throughout the health care system.

Developing an education and assessment tool can be challenging for a competency as seemingly broad as SBP. Varkey et al4 described the use of an OSCE-based assessment following a 3-week quality improvement elective for preventive medicine and endocrinology fellows that allows for the demonstration of skills and process validation. Patterson et al5 used pretest and posttest, and annual oral examination questions to assess the effectiveness of a team- and project-based SBP curriculum. Wang and Vozenilek6 used high-fidelity patient simulation for teaching SBP to emergency medicine residents. Others have successfully used web-based teaching and assessment of SBP, concluding that this method is simple and reproducible, yet maintains construct validity and provides durable learning for a broad range of residents.7,8 

Balmer and colleagues9 found that residents perceive the academic health care setting as separate and distinct from the “real” system in which they will eventually practice, and that residents wanted education specifically in SBP areas. In a recent commentary, Batalden and Leach10 note that it is time to stop “protecting” our trainees from the “system,” but rather that “It is time to incorporate mastery of systems as part of the educational agenda; the well-being of both patients and doctors depends on these skills” (p. 2). Similarly, Graham et al11 suggest developing a formalized taxonomy for SBP education, although actual concrete examples of measuring progress are limited, and existing tools run the risk of measurement pitfalls later. Though we hesitate to admit it, cost is a driving factor of many medical decisions. Residents should be exposed to this reality early.

We designed an education program that included a grand rounds lecture that was intended to familiarize the residents with the overall anesthesia economic climate, as well as web-based didactic sessions focusing specifically on drug costs. To develop a concrete application pertinent to the intraoperative period familiar to all anesthesiology residents, we focused on knowledge of the cost of drugs used during the perioperative period. Prior to the study, the pharmacy had requested a change to less expensive agents (specifically, volatile agents). Some questioned the real or potential impact that this practice change might have on pharmacy budgets compared with changes in patient recovery and safety.

We also designed an assessment tool that challenged residents to design an anesthetic given standard perioperative parameters (such as age, weight, and time). By mandating a maximum dollar limit, residents were forced to integrate factors associated with cost as well as those associated with a “good” anesthetic (stable cardiovascular parameters, adequate anesthesia for surgery, and control of pain and postoperative nausea and vomiting (PONV). The tool allowed us to specifically assess their application of the knowledge to a simulated but realistic clinical scenario, and creates the base for intervention and remediation for residents whose plans are inadequate. It also helps us to further refine our education program as we determine which educational components are effective.

Potential advantages of this type of assessment include:

1. The infinite number of possible scenarios allows for a new test every time it is given. Factors that can be varied and that affect cost include patient age, comorbidities, anticipated hospital stay, and drug allergies. Does the patient have a significant history of PONV? Should an epidural be used? Is ropivacaine a more cost-effective choice than bupivacaine? If so, by how much? Is dexmedetomidine an appropriate choice to smooth emergence? How much does that add to the cost of the anesthetic?

2. Flexibility of case type by specialty. Preoperative testing, intraoperative management, recovery care, as well as pain, obstetric, and critical care scenarios can be easily developed to address cost factors in health care delivery. Again, by assessing baseline knowledge, providing web-based education, and then assessing postintervention gains in knowledge with a typical clinical scenario, SBP education can be tailored to a wide variety of scenarios.

3. Ease of administration. Today's residents are extraordinarily adept at obtaining knowledge from the Internet, and web-based education is a simple tool to reach a large number of residents with a user-friendly interface that is educational, allows work to be done at personal time and place preference, and provides the information in an engaging format. As pointed out in a study detailing the advantages of a web-based SBP model for internal medicine residents, faculty expertise and participation is not a requirement for resident education in SBP.12 As we discovered in our original survey, faculty and residents shared a similar lack of knowledge of drug costs. Although we did not specifically educate faculty or assess faculty knowledge again after our educational intervention, we suspect that our residents had a better understanding of drug costs than the faculty at the conclusion of the program.

4. Reproducibility. Innovative, group-based, project-oriented, SBP-education scenarios have been published for various specialties including anesthesiology,2 radiology,13 obstetrics,5 and surgery.14 Although these studies used novel and broad approaches to SBP education, they were less concerned with reproducibility. Faculty involvement with their approaches was high, assessment methods were subjective, and construct validity was questionable. Judging whether all group members participated equally is difficult, as is comparing the quality of projects. In contrast, we believe that an individual, web-based system allows for an objective and reproducible education program.

In this article we present the findings of the initial survey given to all members of our department, the tools used to assess and improve knowledge among our residents, and then we discuss the competency itself and how anesthesiology programs can incorporate our tools and others into their educational armamentarium.

Following departmental approval, all residents (10 per level of training), certified registered nurse anesthetists (CRNAs), and faculty of the Stony Brook University Department of Anesthesiology were given a survey instrument and asked to complete it without relying on reference materials (or pharmacists). Surveys were completed under direct observation, allowing 10 minutes per participant.

A combination of required didactic education (grand rounds, visiting professors) and web-based education available on a secure departmental website was administered 1 month from the time of initial assessment to the resident participants in the initial survey. The grand rounds program was entitled “Future Trends in Anesthesiology Economics” and included the following topics:

1. What economic issues are faced by anesthesiologists and anesthesia practices?

2. What are the major activities and challenges with current health care financing system for anesthesia service?

3. How will ongoing Medicare changes affect your future practice?

Three months after the initial assessment, resident participants in the initial survey were asked to design anesthetics for predefined cases. They were directed not to spend more than $100 for anesthetic drug delivery. Drug costs used for comparison were obtained from the Stony Brook University pharmacy (table 1). Specific instructions to address intraoperative surgical optimization and postoperative needs such as controlling pain and PONV needed to be addressed within the anesthetic plan. This assessment tool was chosen to allow evaluation of the following tasks: (1) knowledge of drug costs, (2) understanding of the basic anesthesia criteria for a “quality” anesthetic, and (3) knowledge of ancillary costs of care delivery as it may relate to their anesthesia care provided (eg, how PONV can lead to increasing postanesthesia care unit costs).

The survey was repeated 1 year after the initial assessment. The 3 cases designed by the residents and the survey tool are provided as web-based supplemental material to this article.

Six CRNAs, 12 faculty members, and 23 residents (9 clinical anesthesia-1, 7 clinical anesthesia-2, and 7 clinical anesthesia-3) completed the original survey instrument. Our initial survey showed that most practitioners had a poor understanding of anesthetic drug costs, regardless of level of training or degree. A comparison of data for the 3 cases (cases A, B, and C) is displayed in tables 2, 3, and 4, respectively, and intergroup variability to volatile anesthetic cost estimates (by way of example) is shown in figure 1. The variability of responses made statistical analysis difficult, with responses varying by as much as 100 times within and/or between each group.

All residents completed the mandatory online education module, and more than 80% of the initial survey group attended the departmental grand rounds program. All residents completing the repeat survey both attended grand rounds and completed the online assessment tool (n  =  16).

All residents (100%) met the goal of designing an anesthetic for all cases within the $100 limit. Most did not choose the least expensive agent for all cases, but rather altered costs to achieve what they thought was optimal care for the given scenario.

Repeat survey results demonstrated an improvement in nearly all cost estimates with reduction in variability (P < .05, Student unpaired t test, n  =  16), although estimates of volatile anesthetic (cases B and C) and reversal agent (cases A and B) cost did not achieve significance at the .05 level consistently. Participants completing the survey were the same clinical anesthesiology-1 and clinical anesthesiology-2 residents as the prior year (that is, they were now at the clinical anesthesiology-2 and clinical anesthesiology-3 levels of training, respectively). tables 5, 6, and 7 display initial and repeat values for cases A, B, and C, respectively, with P levels. figures 2, 3, and 4 present comparison data for all residents (averaged for both years) as a function of preintervention and postintervention data.

Our results reveal that initial understanding of anesthetic drug costs was low throughout our department, for not only residents, but also CRNAs and faculty. A simple competency-based educational intervention consisting of a lecture plus online teaching tool improved resident understanding significantly within a relatively short period of time. The following discussion will focus on our findings, the SBP competency in general, and future suggestions for incorporating a simple, yet formalized, systems-based module into the current resident education curriculum.

Asking questions and challenging thoughts and ideas of students has been the cornerstone of resident education. In anesthesia education, that frequently takes the form of an abbreviated question and answer session in the operating room, allowing for brief didactic interactions. Although one can certainly argue that the quality of anesthesiologists has been uniformly high over the past few decades, it is more difficult to prove that resident education has been uniformly excellent. In a sense, the purpose of competency-based education is to prove that residents are learning what is being taught, and that the teaching includes not only medical knowledge and techniques, but also the traits that ensure a successful career as a member of a professional medical community.

The current health care climate is complex, and today's residents and students must understand these complexities. Many medical schools have curricula that are embracing the need to develop future physicians with health care resource management education; a 2005 Association of American Medical Colleges report notes that topics such as health care finance (included in 93 of 125 medical schools), health care systems (in 107 of 125 schools), and medical socioeconomics (in 100 of 125 schools) are being addressed within undergraduate education.15 It is not clear, however, how schools with varying curricula are assessing the application/integration of the aforementioned topics.

The ACGME's core competencies help prepare trainees for the challenges of a rapidly changing health care environment. The 6 competencies were developed by a collaboration of educators, residents, and patients, and were narrowed from an original list of 86 statements.16 Some, including medical knowledge and patient care, are self-explanatory and conform to the standard method of teaching and assessments as practiced for decades. Communications skills and professionalism also seem self-explanatory at first glance, but determining just how to teach and assess them can be complicated, and when done properly, require much more effort than a vague evaluation such as “he seems like a good guy.” Practice-based learning and SBP are both difficult to define and teach. The former focuses on the residents' ability to obtain knowledge on his/her own, judge that knowledge and its applicability to a specific patient scenario, and pass that knowledge along to others (ie, teaching students, nurses). Systems-based practice focuses on the resident's ability to understand how the various health care players affect each other in their particular hospital, in their local environment, and in the global medical community, with an emphasis on providing not only excellent and efficient care, but also affordable care whenever and wherever possible. The full ACGME definition states17:

Residents must demonstrate an awareness of and responsiveness to the larger context and system of health care, as well as the ability to call effectively on other resources in the system to provide optimal health care. Residents are expected to:

1. work effectively in various health care delivery settings and systems relevant to their clinical specialty;

2. coordinate patient care within the health care system relevant to their clinical specialty;

3. incorporate considerations of cost awareness and risk-benefit analysis in patient- and/or population-based care as appropriate;

4. advocate for quality patient care and optimal patient care systems;

5. work in interprofessional teams to enhance patient safety and improve patient care quality; and

6. participate in identifying system errors and implementing potential systems solutions.

This SBP competency has placed additional burdens on residencies to develop de novo programs without clear direction from the ACGME. In addition, it is difficult to validate assessment tools within the confines of a single residency, especially when no validated tools are published. The competency appears more applicable to primary care specialists, who are typically perceived to be the front-line patient advocates in the complex medical system environment. Although anesthesiologists are considered perioperative physicians, our impact on health care is primarily focused on the intraoperative experience. There is minimal opportunity, given our truncated patient interaction, for such things as teaching of health care delivery systems or assessing hospital-wide or global impact of our intraoperative actions.

We believe that as a specialty we should develop a systemic and formalized concept of SBP. With our experience in the perioperative arena serving as a starting point, we should develop a plan to better integrate our care into the bigger health care continuum. Key factors should include:

1. Costs of anesthesia care. Knowledge of common costs for anesthesia drugs, equipment, and procedures can be taught and integrated into daily practice, as can the impact of perioperative pharmacy costs on total hospital pharmacy budget. The impact of anesthesia care on total inpatient and outpatient charges can also be easily obtained from published literature and in-hospital systems, and can also contribute to a further understanding of anesthesia costs as a part of a greater health care system.

2. Understanding professional practices as they relate to anesthesia. For example, teaching efficient perioperative throughput, with a focus on how cases are booked (block time, nursing shifts and room closures over the course of a day), and how patients flow from preoperative area to discharge (with associated potential bottlenecks), is a specific practice-management issue that can be developed in a formalized program.

3. Partnering with health care providers. Ordering appropriate consults and correctly ordering and evaluating diagnostic tests are other pathways to integrating SBP education. Teaching and then assessing a resident's understanding of these factors can demonstrate their proper integration and application of this education.

Learning current acquisition costs of drugs is not a perfect measure of SBP (2 drugs in our survey, for instance, have become available as cheaper generics since we began our work), and the relationship between hospital costs and patient charges are complex and convoluted. Nevertheless, we believe that teaching residents to evaluate cost-effectiveness when making clinical decisions will aid them in their future practice, and is at the heart of SBP intent. Englander and colleagues18 have shown the true potential of SBP cost-based education with an estimated savings per year of over $500 000 to their health care system following implementation of a program that taught pediatric residents the economic differences between laboratory and point-of-care testing. More recently, Graham and colleagues11 have suggested an SBP taxonomy with the resident filling multiple roles in the greater health care setting, including that of “resource manager,” requiring the trainee to demonstrate cost-effective practice by performing cost-benefit analyses, using resources appropriately, and providing quality assurance based on appropriate resource allocation.

It is possible that the reduction in variability in nearly all resident cost estimates may not be due solely to our educational intervention. The time between assessments might have allowed the residents to learn about anesthetic cost through daily experience, independent research/investigation, or maturation of anesthetic practice. It is our belief, however, that given the improvement in accuracy and the reduction in variability, it is likely that the collective cohort improved to the extent reported because of the educational intervention. We also acknowledge that our statistical process may be viewed by some as suboptimal because of our small sample size and large individual variations in responses. These factors made more detailed statistical analysis difficult, and further attempts at statistical manipulation could have confused our results.

The most important implication of our study is that anesthesiology residents can be taught SBP, in this case the implication of their practice choices. Anesthesiology training programs are charged by the American Board of Anesthesiology to train consultants and not technicians. Residents understand that the system in which they train is distinct from the “real system” that they will be practicing in after graduation. By integrating a formalized cost analysis of our specialty into the education program, we are starting to address not only the SBP competency of the ACGME, but also the bigger picture of how anesthesia care fits into the local, regional, and global health care economy, and how a practitioner's individual decisions will affect his or her practice and health care as a whole.