Assessing Entrustable Professional Activities Using an Orientation OSCE: Identifying the Gaps Open Access

The beginning of residency provides challenges for new trainees, forcing them to quickly integrate into a new learning environment and independently apply clinical skills acquired during medical school. Many people assume that trainees are ready for this transition and layers of supervision will compensate for a lack of skill and experience.¹  While it is impossible to create a uniform curriculum for all medical schools, the Association of American Medical Colleges (AAMC) describes 13 Core Entrustable Professional Activities for Entering Residency (CEPAERs) every graduate should be expected to perform proficiently on the first day of residency.^2–4  Each training program's intern cohort is comprised of individuals from different medical schools that place varying levels of emphasis on CEPAER development and assessment.

Studies have reported the value of baseline skills assessment during intern orientation using an objective structured clinical examination (OSCE) based on Accreditation Council for Graduate Medical Education (ACGME) core competencies.^5,6  Though guiding principles for implementation of the CEPAER framework have been published, studies using this framework to assess intern baseline skills during orientation are limited.⁴  In spite of these efforts to develop and use Entrustable Professional Activity (EPA)–based frameworks in undergraduate medical education, residency program directors (PDs) continue to express concerns about the preparedness of medical school graduates.^7,8  Studies have shown wide variability in PDs' confidence in interns' ability to perform CEPAERs.^7–9 

There is a gap between PDs' perception of intern abilities and actual intern abilities. Given these challenges, orientation should evolve to address more than the administrative needs of interns. We propose that a CEPAER-based OSCE during orientation is a feasible way to assess incoming intern skills. The overall aims of this assessment were to (1) assess individual intern's baseline clinical skills and provide formative feedback; (2) determine an intern's readiness for residency; (3) inform individualized education plans; and (4) address identified gaps through curricular changes. Here we describe the design, logistics, and results of this formative assessment program.

The OSCE day was implemented during intern orientation in June 2017 at the University of Iowa Hospital and Clinics. Based on a literature review and an internal needs assessment (provided as online supplemental material), the Intern Orientation Committee developed stations (cases and faculty global assessment forms) linked to CEPAERs essential for a successful intern year.^8,10  The six 20-minute stations (15-minute simulation followed by 5-minute faculty feedback) evaluated 8 EPAs (figures 1a and 1b). Standardized patients (SPs) were trained by simulation center staff and core faculty for stations 1, 2, 5, and 6. SPs completed the University of Iowa Carver College of Medicine's communication checklists with validity evidence for this setting and subjects following each encounter (provided as online supplemental material). Five graduating students participated in a pilot study to determine feasibility and identify resource needs (provided as online supplemental material). Case modifications and additional training occurred based on pilot results.

Thirty SPs and 18 core teaching faculty served as evaluators (provided as online supplemental material). The intern cohort was divided into morning (17 residents) and afternoon (16 residents) sessions. All stations ran simultaneously in 3 clinical suites (provided as online supplemental material). Prior to the event, station leaders trained faculty evaluators on global rating forms (provided as online supplemental material). Encounters were recorded, and immediate faculty feedback was provided at 5 stations. After 4 months, interns received a written performance summary based on faculty and SP evaluations (provided as online supplemental material).

The cohort was comprised of 33 interns from 19 medical schools pursuing 3 specialties. All participants completed a self-assessment of baseline skills and a post-OSCE survey (provided as online supplemental material). The average cost was $191 per intern (table 1), and the faculty time for training and assessment was estimated to be 10 hours.

This project was deemed non–human subjects research by the Institutional Review Board of the University of Iowa.

Faculty ratings among the 6 stations were compared using the Friedman test, with post-hoc pairwise comparisons based on Friedman rank sums. Using the definition of percentage of total points ≥ two-thirds for meeting criteria (based on the Intern Orientation Committee's recommendation), the percentage (95% confidence interval [CI]) that met criteria for each station was analyzed. Analysis of the outcome of met criteria was compared among components using a generalized linear mixed model with logit link function. The Tukey-Kramer test was used for post-hoc pairwise comparison between components. Prior training on selected CEPAERs was analyzed by reviewing post-OSCE survey responses.

Analysis of the stations based on percentage of maximum score showed “nurse call” (median score 92, interquartile range [IQR] 75–96) was the highest-performing station. This was significantly higher compared to “informed consent” (median score 73.5; IQR 57–90; P = .038), “handoffs” (median score 71.5; IQR 64–81; P = .008), and “SOAP note” (median score 71.5; IQR 51.5–80; P = .0001). “Oral presentation” had the second highest score with a median score of 83 (IQR 67–92), also significantly higher than “SOAP note” (P = .004, figure 2a).

Analysis of the stations based on met criteria (figure 2b) showed there was an overall significant difference among stations in the proportion that met criteria (P = .006). The 2 stations with highest percentage that met criteria were “oral presentation” with 94% (95% CI 0.78–0.99) and “nurse call” with 91% (95% CI 0.75–0.97).

Prior exposure to some EPAs was assessed in survey data (figure 3a). Interns performed lower in skills (handoffs and informed consent) for which they did not report formal training in medical school.

The post-OSCE survey (figure 3b), completed by all 33 participants, indicated the majority of interns (79%, 26 of 33) viewed the skills assessment as useful, helping them reflect on their strengths and weaknesses prior to starting training. Immediate faculty feedback was cited as the most beneficial part of the exercise (94%, 31 of 33). The post-OSCE survey comments were organized by 3 themes: (1) self-reflection and self-directed learning; (2) feedback; and (3) relevance, value, and efficacy of experience (figure 3c). During debriefing, most faculty commented on their positive OSCE experience.

Following the CEPAER-based OSCE, data were compiled and used for curricular changes and early individualized feedback (table 2). Formal instruction was incorporated into didactic sessions (blocks of time in ambulatory curriculum and yearly workshops) for the lowest-performing stations. A longitudinal palliative care curriculum was designed and implemented for “difficult conversations.” Structured feedback sessions between interns and senior residents were incorporated into the night float curriculum. This allowed for real-time feedback of EPA skills related to “handoffs” and “SOAP note.” In addition, handoff training now includes simulation, direct observation, and monthly reinforcement of best practices. Individual performance reports were provided to interns and used during 6-month evaluations with PDs (provided as online supplemental material). However, these reports were not used for any promotional or advancement decisions. Lower-performing interns received individualized coaching from chief residents and core faculty early in their residency.

An OSCE-based assessment of CEPAER skills during intern orientation demonstrated that many interns performed below expectations for handoffs, informed consent, and writing SOAP notes. They performed above expectations in oral case presentations and handling nurse calls. The intervention cost averaged $191 per intern and required 2 half-day sessions. This formative assessment was well-received by faculty and interns. The OSCE results informed subsequent individual skills development and curricular changes.

There is agreement in the medical education community that medical students should enter residency with a minimum level of competence in certain domains of knowledge and skills. Previous studies have used OSCEs based on ACGME competencies to assess interns' baseline skills.^5,6  In contrast, we used the AAMC CEPAERs to assess baseline skills of incoming residents. Furthermore, to the best of our knowledge, our study is the first to actually use results from the OSCE to make real-time curricular changes and provide individualized feedback.

A recent study suggested that PDs across all specialties believe there are several EPAs that interns cannot perform without direct supervision.⁸  Similar findings were reported in surgery and internal medicine.^7,9  These studies were based on PD or resident perceptions, using survey data.^7–10  Our results are based on actual performance data, adding objective evidence to these reports and documenting a gap between what is expected and baseline performance. Handoffs and informed consent were our lowest-performing EPAs, consistent with historical PD perceptions.^7,8  Lack of practice, combined with little guidance and feedback, may make it difficult to master these tasks. Not surprisingly, in our study, the performance on “handoffs” and “informed consent” correlated with the reported lack of past formal undergraduate teaching. Lower performance on documenting a clinical encounter may correlate with the introduction of the electronic health record. Limited opportunities for documentation reduces medical students' experiential learning and feedback.¹¹  If these EPAs are reasonable expectations for medical school graduation, we need to consider how undergraduate curricula can meaningfully incorporate them and develop a consistent framework for longitudinal observation.² 

We learned several lessons during implementation of this exercise. First, the pilot session provided valuable information regarding the design and logistics of the OSCE. Second, the assessment necessitated significant faculty and facility time. Finally, the interns' desire for immediate feedback was a common theme and emphasized the importance of capitalizing on teachable moments.

This study has limitations. It was implemented in a single program, limiting generalizability. This exercise introduces new OSCEs, which have limited validity evidence to date. Although the rigorous development process and favorable intern and faculty impressions speak to the strength of the assessment, additional validity evidence is desirable. We observed that the range of scores at different stations for an individual intern was high, and it is not clear whether this variance represents individual abilities or different emphasis on CEPAERs at various medical schools. We surveyed the interns regarding formal training/exposure related to these CEPAERs, but did not ascertain the extent to which they participated in capstone courses with emphasis on the CEPAERs.^12,13  While we have considerable experience using OSCEs for medical students, our experience with residents is limited.

This OSCE exercise as presently constructed is intended for formative assessment only; therefore, it cannot be used as a high-stakes assessment until more data to determine validity and reliability are available. Additional experience with intern cohorts at other institutions will be important to determine if CEPAER-based OSCEs during intern orientation provide additional value to a fourth-year capstone course. Whether this exercise is feasible across all graduate medical education programs will require further study of its implementation in different specialties.

This OSCE offers just-in-time baseline information regarding critical intern skills. The interns performed below expectations in the areas of handoffs, informed consent, and writing SOAP notes. Post-OSCE survey results showed the majority of interns found this exercise, and particularly immediate feedback, useful. Based on gap analysis, this assessment can drive individualized feedback as well as continuous curricular improvement.