Best Evidence Calling for Educational Reform: Will We Do the Right Thing?

In this issue of the Journal of Graduate Medical Education, Wood et al1 present their work entitled Neonatal Resuscitation Skills Among Pediatricians and Family Physicians: Is Residency Training Preparing for Post-Residency Practice?1 Their report analyzes 302 completed surveys from practicing primary care pediatricians and family practitioners, 22% of whom were practicing in rural areas and 78% were practicing in urban/suburban areas. The pediatrician respondents reported having more experience with attending deliveries and neonatal resuscitation skills during residency training than family medicine physicians, an expected result given the more prescriptive Accreditation Council for Graduate Medical Education (ACGME) pediatrics requirements for resident experience in the neonatal intensive care unit. The study found that three-quarters of pediatricians and family medicine physicians reported they had not attended a delivery in the year prior to the survey. Rural physicians, however, were significantly more likely to attend deliveries than their urban/suburban counterparts. The most worrisome finding is that among rural physicians who had attended any deliveries in the last year, 7/16 pediatricians (44%) and 10/25 family physicians (40%) felt inadequately prepared for at least one delivery.

Based on information the authors present, we would answer the question in their title (“Is residency training preparing for post-residency practice?”) with a probable “no.” The findings are an indictment of the current “one size fits all” approach to residency training. As the study illustrates, training experience does not always align with authentic professional activities that practitioners are called upon to perform in their everyday practice. In this survey, either the practitioners were trained to perform skills that only one-quarter of them would actually engage in during routine practice, or, even more concerning, a substantial percentage of practitioners currently engage in practices for which they feel inadequately prepared. The serious nature of the resuscitation skills in question emphasizes the critical need to align residency training experience with authentic professional responsibilities of a practicing physician. Where do we go from here?

The ACGME Outcome Project,2 introduced a decade ago, was intended to shift the focus of medical education and training from the structure and process of training to the desired outcomes of training, to address the problem highlighted in this study. Instead of reviewing programs to ensure that they provided the required months of neonatal intensive care unit training, for example, the ACGME review would focus on whether a program's graduates are competent to resuscitate sick neonates. However, how can we as educators determine if a learner is competent? This question poses one of the greatest challenges in competency-based education. There is currently a paucity of reliable and valid tools to assess competence in a meaningful way,3 so how do we begin to move forward in the face of this daunting challenge? We believe there is a path forward if we learn from outcomes such as those presented by Wood et al,1 and we also learn from and apply the findings of educational innovation efforts currently underway in the Initiative for Innovation in Pediatric Education (IIPE),4 the Family Medicine Preparing the Personal Physician for Practice (P4) project,5 and the Internal Medicine Educational Innovations Project (EIP).6 

To accomplish an outcomes-based approach to medical education that is meaningful, we need to begin developing training requirements by backward visioning. This requires determining the authentic professional responsibilities of practitioners and mapping these responsibilities back to graduate medical education training requirements and subsequently to undergraduate medical education as a continuous learning experience. We know from the work of Balmer et al,7 who performed a qualitative study of the ACGME competencies by directly observing 143 hours of clinical work in the inpatient unit of a large children's hospital, that while the competencies were never mentioned by name, all of the conversations around the care of patients could be mapped to the 6 competencies. The message here is that the competencies come alive in the context of care delivery. As we have witnessed over the last decade, in the abstract, the competencies leave many educators and faculty confused about what they represent and how to implement them in a meaningful way. The work of ten Cate and Scheele,8 in their seminal article Competency-Based Postgraduate Training: Can We Bridge the Gap Between Theory and Clinical Practice?, paves the way toward reframing competencies in the context of clinical care, making the competencies more intuitive for clinician-educators. The authors suggest that communities of practice identify the entrustable professional activities (EPAs) that define their specialty or subspecialty.8 EPAs are defined as the routine units of work in which a physician engages that result in outcomes that can be witnessed or observed. Each EPA requires integration of ACGME competencies. An example of an EPA for a pediatrician or family physician practicing in a rural area is to “resuscitate and stabilize a newborn in the delivery room.” The word “entrustable” is critically important and refers to the privilege of engaging in the activity without direct supervision only after one has been observed successfully performing the activity. Success would be defined here as having the activity, in this case, to resuscitate and stabilize, result in the desired outcome, a stable newborn.

Reinforcing the work of Balmer et al7 and ten Cate and Scheele8 is the recently published Carnegie Report9 on educating physicians, which addresses 4 goals to accomplish the proposed reform needed in medical education. One of these goals is “standardization of learning outcomes and individualization of the learning process.”9 The most practical and meaningful approach to determining these outcomes is to apply the EPAs discussed in the work of ten Cate and Scheele, which place the competencies into a clinical context. To this end, the standardization process involves determining not only the outcomes that reflect routine professional activities of the specialist or subspecialist physician but also the context in which they will practice. In the context of the article by Wood et al,1 family medicine and pediatric residents, after some agreed-upon core training, would differentiate generic generalists into more specific generalists, defined by type of future practice, such as rural versus suburban/urban practice. In the case of generalists who plan to practice in a rural setting, there would be an emphasis on resuscitation skills needed in the rural community as this would be an EPA for generalists working in this context. In this case, entrustment, that is, granting the privilege of performing the skills needed to resuscitate newborns without direct supervision should occur prior to graduating from a residency training program.

Another important step in making the transition to a competency-based system of education is to determine when a learner is competent. To address the challenge of competency-based assessment, the ACGME introduced the Milestone Project.10 Each specialty community is being asked to (1) refine the 6 ACGME competencies in the context of their specialty, (2) develop performance standards expected of different levels of trainees, and (3) identify and/or create assessment tools that can be used at the national level. This work is now under way for many specialties.

Individualization of learning during residency prepares one for the intended practice that one envisions at that moment in time; yet what happens when careers evolve and the practice environment changes? This suggests the need to embrace the competency of practice-based learning and improvement for life-long improvement during training and practice. The American Academy of Pediatrics has been involved in The Physician Reentry into the Workforce Project,11 which would be helpful to those who are making dramatic career changes and need to learn a substantial new body of knowledge and skills. For those whose careers evolve and who therefore need to acquire a more limited body of knowledge and skills along the way, ongoing quality improvement may be the answer. The process of practice-based learning and improvement will lead to identifying gaps, new learning, applying new learning, and measuring improvements. In addition, as Maintenance of Certification evolves and expands the breadth of learning opportunities, pediatricians and other specialists may be better able to address their specific learning needs for new knowledge and skills through this process.

As pediatricians, we can speak to the steady progress of our specialty in development of clinically based competencies, individualization of training, and successful practice-based improvement behaviors that are sustained after graduation. Our hope is that this work will bring us closer to the goals of improving our current methods of assessment for judging competence and ensuring that these competencies are relevant to the future practice of physicians.12,13 As practitioners of a self-regulating profession, we must learn from the evidence presented in the study by Wood et al1 of a mismatch between training and practice. We believe that the incorporation of EPAs and Milestones into the foundational principles of medical education and training can reduce this mismatch and thus is the right thing to do for our trainees and our patients.