Centralized Assessment in Graduate Medical Education: Cents and Sensibilities

The start of residency marks the beginning of practical training for all new physicians. It often means a new learning environment and marks a significant increase in responsibility for patient care. A dominant assumption has been that our system of education prepares transition-ready learners and that appropriate layers of supervision are in place to ensure the safety of patients.1–4 However, a close look at the transition from medical school to teaching hospital can reveal a different reality. Entry into residency is characterized by an “educational handoff” from an institution with a centralized process–the medical school–to another institution, which often offers a decentralized form of “on-the job training” as the primary mode of education: the teaching hospital. The pace and complexity of even highly supervised patient care requires that new residents possess basic skills and the ability to apply them in their new surroundings. The financial pressures for billing can limit medical student experiences in the clinical arena, and their clerkship and elective experiences are variable.5–7 At this stage in their education, new residents have survived a gauntlet of standardized and multiple-choice examinations; however, even with the implementation of the United States Medical Licensing Examination Step 2 Clinical Skills, the amount of assessment at the “shows how” or “does” level varies among graduates of different schools and among individual learners.1,8,9 The first postgraduate year (PGY-1) is all “does” as residents assume responsibility for new patients in a new hospital, a new city, and for some, a new country.8 Throughout the nation, July 1 marks a critically important transition for physician learners into US hospitals.

Hospital orientation has a long tradition of “force feeding” new residents, or asking them to consume copious amounts of crucial information from a “fire hydrant” before starting direct patient care.10–12 In fact, ever-increasing amounts of information are being packed into the days and weeks prior to the official first day of residency. Historically, this information has been provided in lecture format, with little hands-on application, and with the assumption of adequate knowledge and skill acquisition.13 What has not traditionally occurred during those days and weeks is a baseline assessment of the actual patient care skills that will enable new trainees to survive and succeed in their new roles.14–16 Accrediting organizations and influential bodies such as the Institute of Medicine17 and Medicare Payment Advisory Commission18 have called for sweeping reforms of the educational and assessment enterprises in health care. Some of the primary focus areas are the improvement of patient safety, assessment of both learner and patient outcomes, and the use of those outcomes to inform continuous quality-improvement efforts.

The current medical education system exists in 3 segments that have not been well linked. Consequently, influential groups have called for more consistent educational processes and outcomes assessments. Undergraduate, graduate, and continuing medical education are all being challenged to break out of their silos and make intentional connections between their curricular and evaluation components. The long-entrenched “dwell time” methodology is being replaced by a competency framework. In this new paradigm, the exact role of the centralized graduate medical education (GME) office in assessing resident outcomes remains unclear. Although resident assessment has traditionally resided at the program level, more and more accreditation institutions, such as the Accreditation Council for Graduate Medical Education (ACGME) and The Joint Commission, are beginning to require some institutional knowledge of outcomes/performance at the resident level.19 

These new requirements have sparked many institutions with the responsibility for residency sponsorship to seek new methods to ensure accountability to their accrediting bodies, their trainees, and their patients. These institutions must “assure that each sponsored program provides a curriculum and an evaluation system to ensure that residents demonstrate achievement of the six general competencies” and that residents develop a “personal plan for learning to foster continued professional growth.”20 Although it appears that centralized assessment may have a growing role within medical education, it is not clear how institutional resources can ensure continuous improvement at the resident level or who will assume accountability for these efforts. According to Krupat and Dienstag,21 “assessment should always be designed to enhance student learning and to facilitate a comfortable learning environment…. (assessment) allows faculty the opportunity or, more accurately, allows them to fulfill their duty to students–to identify marginal performance and to provide support and remediation.” We argue that starting this process in the early part of training is germane to creating an environment of continuous learning and patient safety. It also allows for enhanced documentation of competency assessment and education. It also may allow a broader platform for establishing valid and reliable assessment methods.

In this article we will describe how 2 institutions in the state of Michigan responded to these challenges—how to initiate a cycle of assessment and feedback for new residents that assures (1) rapid identification of new resident weaknesses for the benefit of patients, the resident, and their program directors; and (2) seeks to accomplish the larger goal of informing change in both undergraduate and graduate medical education. We list “Lessons Learned” and “Critical Principles” to provide touch points for discussion among stakeholders at institutions considering the development of similar experiences for their trainees. Finally, we will review the promise and problems related to such centralized assessments.

For the past 7 years (June 2002 to June 2009), the University of Michigan Health System has implemented an intern Objective Standardized Clinical Examination (OSCE)–the Postgraduate Orientation Assessment (POA).22,23 This entering assessment has provided program directors in 14 different specialties with baseline performance data on their incoming interns. Over these past 7 years there have been several modifications to the 8 to 10 station examination that to date has included assessment of informed consent skills, aseptic technique competency, sociocultural communication (via standardized patients), evidence-based medicine skills (developing a clinical question/choosing the best evidence), pain recognition and treatment, fire safety skills, geriatric functional status, patient safety (order writing and/or patient handoff), radiographic interpretation, and critical laboratory value recognition and response. Topics were chosen based on a previously developed OSCE administered to the medical students at the University of Michigan Medical School, with a focus on skills likely to be needed by residents in the first 4 months of their training. These stations used various forms of education and assessment modalities, including standardized patients, PowerPoint presentations, videos, and web-based programs. Remediation for each station, usually in the form of take-home literature, was provided at the time of the assessment. In this test we have tended to focus on the institution's need to cover core Joint Commission patient safety goals. We theorized that interns' exposure to these important areas of hospital function outside the POA is limited, and thus we moved to fill this gap with a centralized assessment.6,24,25 

In June 2006, program directors of Michigan State University College of Human Medicine-sponsored residencies, led by the Associate Dean for GME, came together to respond to a mandate for performance-based assessment of key, patient safety-related competencies desired of early PGY-1 trainees.26 The overarching goal was to develop stations requiring new residents to demonstrate, at the “shows how” level, that they could perform specific patient safety-related skills. Stations employed a combination of standardized patients or team members, partial task trainers, patient mannequins, or in some cases multiple simulators to achieve the relevant objectives. A list of over 20 potential stations was whittled to the final 10 by group nominal process among over 50 stakeholders from multiple institutions. Program directors, medical and nursing education faculty, hospital administrators, and quality assurance officers all participated in this process, and this enabled us to promote buy-in and create stations relevant to all stakeholders.

At both the University of Michigan Health System and the Michigan State University College of Human Medicine, multiple contributors came together to “blueprint” their PGY-1 OSCEs (table). National mandates and goals, coupled with local process and priorities, resulted in examinations with high degrees of relevance for all stakeholders.

• █ Early buy-in by multiple stakeholders is critical to the success of a centralized assessment that meets the needs of multiple accrediting agencies and the residency programs.

• █ Begin with a needs assessment.

• █ Garner institutional/stakeholder buy-in (program director input is crucial).

• █ Blueprint your OSCE.

• █ Secure funding (budget allocation for an annual event).

Indeed, the implementation of a health system-wide OSCE is costly. The POA requires at least 1 additional day of paid orientation. Residents are scheduled in groups of 20 to 30, to make the grouping as specialty-specific as possible. We have the advantage of using the University of Michigan Medical School Learning Resource Center and Standardized Patient Facility. The Learning Resource Center is in proximity to the hospital and has long been used as a site for multistation testing, as is done with both second- and fourth-year medical students at our institution. Institutions without a standardized patient center may consider developing low-tech models as long as they contemplate the standardization of their assessment measures and models in advance. Although many institutions have developed low-cost computerized orientation materials,11 including pretesting and posttesting and methods for assessment of clinical competency can provide a unique opportunity to assess baseline understanding of core concepts.

Following the administration of the POA, both interns and program directors receive individualized score reports. This provides the interns a score on each station with a comparison to that year's mean; the program director receives that same information on the individual, but also receives all the scores for their program and various comparison graphs of their program versus others in the institution. In addition, they receive year-to-year performance on each station and the same remediation materials provided to the interns. This documentation allows for each inclusion into ACGME program information forms and/or requested appendix for a subsequent site visit.

The University of Michigan Health System is a large institution with 83 ACGME-accredited programs in various fields. The health system includes a medical school with facilities to accommodate a large-scale assessment like the POA. Nevertheless, we have found that many of the modules at the University of Michigan Health System are portable and exportable to other institutions, often at a much smaller scale. Centralized assessment offers an opportunity not only for regional and national sharing of materials but cross-institutional development. These efforts can often reduce the cost and allow for development of affordable assessment modules that anyone can implement. For example, we have shared some of our modules with at least 2 community hospitals that have implemented similar assessment formats.

The College of Human Medicine is fortunate to have access to the Michigan State University Learning and Assessment Center, located on the East Lansing campus. Through a cooperation of the 4 health colleges at Michigan State University, it provides the infrastructure for simulation and assessment, which enabled us to bring the PGY-1 OSCE to fruition. Help with scenario development, patient training, simulation, and technical expertise were all provided by the director and her staff. All OSCE stations are digitally video recorded, and both standardized patient ratings and trainee products are computer-entered and recorded in a unique electronic file. This enables all performance data to be accessed by the trainees, program directors, and institutional officials within 1 to 2 days. The first patient safety OSCE took place in 2006 and had 235 participants. Two of the 10 stations provided immediate feedback to the participants via standardized nurse. Although overall feedback from the participants was excellent, the 2 stations that incorporated immediate feedback were thought to be the most useful, and for the next offering, every station was redesigned to provide immediate feedback either by the standardized nurse, standardized patient, or an exemplar, which the residents could access after submitting their postencounter task. Approximately 210 residents from across Michigan have participated in each of the 2007 and 2008 offerings. A “Pediatric Track” has been developed and a new station will be added for 2009 using a mannequin that can be programmed with abnormal physical findings. This strengthens the emphasis on “shows how” by challenging participants to accurately identify common vital signs and lung and heart abnormalities without any cueing from written descriptions.

• █ The presence of a dedicated assessment enterprise is a great advantage, as the logistics of administering these examinations are complex. In the absence of a local center, institutions may consider pooling expertise and assets to develop a regional offering. In addition, the opportunities to share modules across institutions can dramatically reduce the implementation cost and administrative challenges.

• █ Programs with limited access to these types of resources should think carefully about standardization metrics related to assessment.

• █ The format of score reports must be considered and easily formulated to meet accreditation needs.

The following must be considered before establishing an assessment:

• █ Decide whether the examination will be formative or summative.

• █ Determine how you will provide results to trainees and program directors.

• █ Determine the remediation strategies.

• █ Explore how you will gather feedback from trainees.

• █ Determine the best structure for continuous improvement of the examination.

Between 2002 and 2008 we assessed 1 083 interns during the hospital orientation. Of these interns, 70% stated that they learned a new skill during the POA, 80% thought is was a useful way to spend orientation, 78% felt better prepared for some aspects of internship, and 80% would recommend it for the following year's interns. Overall scores have remained relatively stable since 2003; there is an overall mean score of 73.4 with a range of 69 to 75 over the past 7 administrations. Most program directors review the POA scores with their interns and include their performance in the first semiannual evaluation compilations. Given recurrent inconsistent performance, some programs have implemented curricular changes to include discussion of informed consent procedures, radiology reviews, and core concepts in evidence-based medicine in their journal club offerings.

Over time, the scores on the POA have been relatively constant. This may be attributable to our testing of information, for which many interns have some knowledge at a basic level but have not fully mastered. Also, the constant scores could be a possible limitation of the examination in that it is unable to detect differences in performance over time. Nevertheless, published and unpublished data suggest that University of Michigan Medical School graduates are rarely performing any better 9 to 12 months later on similar stations taken as a fourth-year high-stakes assessment,22 and unpublished data on University of Michigan medical school graduate performance on the Resident Post-graduate Orientation Assessment (POA) and United States Medical Licensing Exam (USMLE). In addition, we found no significant differences between surgical PGY-1 interns and surgical residents in PGY2/3 interns in their performance on the aseptic technique station (t₍₅₅₎  =  0.84, P  =  .404). Similarly, no significant differences were noted in performance between surgical PGY-1 interns and surgical PGY-2 and PGY-3 interns on the informed consent station (t₍₄₉₎  =  0.12, P  =  .902). In contrast, when residents from all specialties were included, PGY-2 and PGY-3 interns showed better performance than the PGY-1 interns on the informed consent station (t₍₅₁₎  =  2.46, P  =  .017).27 

During the 2008 academic year, the University of Michigan Health System undertook a standard setting exercise with 12 faculty, 2 standardized patient educators, and 1 patient. This working group established new cut scores (pass/fail scores) that will better inform program directors on the meaningfulness of these formative scores. These news standards were implemented during the June 2009 administration. Later in the year, we will query the program directors to determine the utility of having similar information for a formative assessment.

Over time, we have crafted the POA to meet the “new” learning needs of our interns. We have piloted a hand-washing quiz, a script concordance test on cultural competency and patient handoff station in the 2009 administration. We continue to work closely with the assessment content team and program directors to ensure that the test sufficiently collects relevant and useful information on incoming intern performance and skill level. We will also continue to work together as an institution to determine appropriate methods for remediation and retesting. In the past, these components have been limited by funding constraints.

Data from the 2006 administration were aggregated into 1 of 2 domains: communication/interpersonal skills and patient care skills. Residents performed much better in the communication skills domain, scoring close to 90% overall. They performed considerably less well on the patient care items, scoring approximately 60% overall. Within that domain, they did best when the assessment related to medical knowledge, and poorest when the assessment involved a psychomotor skill (performing aseptic technique). Although stations were specifically designed to present tasks expected of new PGY-1 trainees, there were significant performance deficits noted, and total scores were highly variable. From the perspective of new resident well-being, the presence of a skill deficit undoubtedly adds to the already highly stressful transition they are making. From a patient safety perspective, these results suggest the necessity for improving the preparation of new residents, and the need for careful supervision and the rapid remediation of deficits. In 2006, a small number of residents repeated the OSCE at the end of their PGY-1 year. Significant improvement was attained in 6 of the 10 stations, but total scores again showed wide variation.

With each administration of this OSCE, participants have expressed a high degree of satisfaction with the experience—especially since the addition of immediate feedback after every station. With successive offerings, increasing numbers of program directors and institutional officials have incorporated OSCE results into early progress sessions with their residents, and into the development of personal learning plans for the first several weeks of residency. One example of curriculum development that has been a direct result of the OSCE is a hand hygiene and aseptic technique teaching and assessment module that is currently being piloted for undergraduate medical and nursing students as well as PGY-1 residents and new staff nurses in our community. The OSCE has served as an impetus for curricular evaluation and reform in our medical school as it has shaped the orientation experience for every new resident in the Lansing community. A new Office of College-Wide Assessment has been created to pay specific attention to the rational assessment of competency across our undergraduate, graduate, and continuing medical education enterprises. Its ultimate goal is to ensure that transition-ready learners are truly prepared to thrive at the next level of responsibility.

• █ Centralized assessment of core competencies enables economies of scale in this age of outcome assessment.

• █ Medical educators should be tapped to facilitate blueprinting the examination, designing the station components, and analyzing the results.

• █ Responding to the results, providing feedback, and planning and implementing remediation are not easily centralized, but require dedicated program director and residency faculty time and resources.

• █ Leadership at the institutional and program level must agree that responding to the results in a timely and effective manner is important. This is a place at which “accreditation accounting” may work to align incentives in very positive ways: when The Joint Commission, Institutional Review, and Residency Review Committee mandates line up, requirements can be satisfied in ways that are directly beneficial to trainees and patients. Institutional leadership must support the response to centralized assessment results as a broad benefit, assuring compliance with accreditation standards while improving the educational program and, hopefully, patient safety.

• █ The examination content may require modification with successive offerings, and should respond to stakeholder feedback.

• █ The assessment at the “shows how” level reveals significant deficits in new resident performance.

• █ The response to and remediation of those deficits requires program director and faculty time and commitment.

• █ The development of standards in performance-based education is a novel concept to many in GME.

It is our intention to continually adapt these intern OSCEs to meet the needs of all stakeholders, and to maintain an emphasis on the link between performance assessment and patient safety. The positive feedback from trainees, especially as it pertains to self-reflection, self-assessment, and areas of improvement, propels us to envision a series of performance-based, formative experiences that help trainees at every transition feel confident in their abilities to do what they desire most–provide the very best, safest patient care. Institutions may solve the challenges of funding such assessments through centralization; however, they must also provide program directors and faculty with sufficient time and financial resources to do what is best for their trainees and the patients under their care.

• █ Centralized assessment of core competencies is important to institutional sponsors of GME programs, program faculty, and trainees at stages of transition to increased responsibility for patient care by revealing performance deficits that must be addressed in the early months of residency.

• █ Deficits found during these assessments have the potential to compromise trainee well-being as well as patient safety.

• █ In a climate of increasing resource constraints, GME programs must exploit “economies of scale” and redefine the roles of program directors and faculty during this critical time.

• █ Continuous review of the offering and awareness of stakeholder priorities is required to maintain maximal relevance.

• █ Funding is required for both the centralized assessment event and for the individual program response.

• █ Results should inform programmatic decisions on educational offerings and level of supervision in the first months of residency.

Assessing core competencies at the “shows how” level of accountability requires time, financial resources, medical education expertise, and a cooperative stance among multiple stakeholders. This may occur within, and in the case of the College of Human Medicine and others, across institutions, but it can be accomplished successfully. Centralized assessment can inform medical educators of strengths and weaknesses within existing curriculum and evaluation efforts, and help them plan necessary interventions based on aggregated performance data. Economies of scale are employed when the competencies assessed are considered critical for all residents. Much more challenging, however, is the translation of individual resident performance results into the ongoing assessment activities of each residency program, and planning for this ideally should be considered as part of the implementation. Centralized assessment simplifies document compliance with accreditation standards; however, the information must be used in the daily work of evaluation in order to be truly innovative.

Centralized assessment at transitions informs the medical education enterprise in 2 directions and can work to strengthen the continuum. It can inform specific, necessary changes in undergraduate medical education when skills assumed to be “well in hand” are demonstrated to be lacking. It informs early months of a new educational program by identifying some critical educational and assessment components necessary at the transition to supervised patient care. Centralized assessment can, in the aggregate, inform the continuous improvement of our educational efforts as it enables individuals to not only remediate their weaknesses, but celebrate their accomplishments using “shows how” data.

The same principle could be used to design a “readiness for practice” OSCE. Such an exercise would inform GME of certain deficits and continuing medical education of imperatives for training and assessment in the transition to unsupervised care. In fact, a culture of lifelong learning and of ongoing “personal learning plans” would be supported by a series of such experiences. The expectation that all physicians, at all points on their professional journey, need to assess their strengths and weaknesses and plan for improvement fits nicely with the “continuous quality improvement” efforts ongoing throughout health care. The possibilities for centralized assessment are vast, and it could be incorporated into the recredentialing process of hospitals or into local continuing medical education activities. If centralized assessment offerings are expanded to faculty, the same lessons learned and critical principles discussed earlier will need to be considered.

Centralized assessment of core competencies, while challenging, has made good sense for our 2 very different institutions. The “cents” required to capitalize on its promise must be provided to program directors and faculty if we are to potentially impact patient outcomes. The feasibility and logistics must make “sense” to all the stakeholders involved in order to ensure success in the area of centralized assessment.