Motivations for Entrustable Professional Activity Assessment: Gaps Between Curriculum Theory and Resident Reality

Competency-based medical education (CBME) represents a global paradigm shift in medical training.^1-4  In 2017, the Royal College of Physicians and Surgeons of Canada implemented a hybrid of CBME adapted to the Canadian context called Competence by Design (CBD), which combines elements of time-variable and outcomes-based approaches to training.⁵  Learner progression in CBD is captured by discrete markers of competence called entrustable professional activities (EPAs). This model integrates Van Melle and colleagues’ 5 core components of CBME curricula: outcome competencies, sequenced progression, tailored learning experiences, programmatic assessment, and competency-focused instruction.⁶  The first 4 of these components in CBD are standardized by the Royal College whereas competency-focused instruction may vary by institution and thus represents a target for closer scrutiny and evaluation.

The success of competency-focused instruction hinges on the motivation of residents to carry out the activities and procedures of new assessment practices. Review of the existing literature on resident perspectives toward CBME and EPAs illustrates that residents embrace the idea of receiving regular feedback in the form of multiple formative assessments such as EPAs completed in the workplace^7-9  and appreciate having EPAs as a clearly defined roadmap leading to graduation and independent practice.⁷  Conversely, others criticize CBME, citing “cherry-picking” of clinical experiences and assessors to achieve the maximum EPA score (otherwise known as “gaming the system”)^8,10  and the administrative burden of EPA form completion.^8,9,11,12  The resident experience with CBD in Quebec, Canada, highlighted the stress and anxiety created by the process of EPA acquisition and completion.¹⁰  This was further corroborated by a 2022 Royal College-led survey in which 73% of residents indicated that the transition to CBD had a negative impact on their health and wellness.¹³ 

This conflicting evidence suggests it is still unclear what motivates learners to obtain EPAs, and there are very few if any studies that have examined this issue. In their seminal review of contemporary theories about motivation to learn, Cook and Artino described self-determination theory, in which learner actions are driven by “intrinsic interests or by extrinsic values that have become integrated and internalized.”¹⁴  Intrinsic motivation has been shown to improve learner performance and well-being.¹⁵  Therefore, in a curriculum where the underlying philosophy of assessment is to drive learning, it is vital to uncover these motivations so that interventions can be devised to increase the active engagement of learners to foster the skills of lifelong learning. Developing lifelong learning skills is anticipated to contribute to improved patient care outcomes specific to meeting the ever-evolving health care needs of the community.¹⁶ 

Our study aims to explore 3 specific aspects of competency-focused instruction in CBD: the motivations underlying resident EPA requests (phenomena in the early stages of exploration in existing literature), perceived importance of EPAs, and furthering our understanding of the process of EPA acquisition in the Canadian context.

Participants in this study were current residents in a variety of specialty programs at the University of Alberta in Edmonton, Canada, who have either implemented CBD or are in their pilot year of implementation of CBD. The local Postgraduate Medical Education Office developed a CBD toolkit to assist program leads, faculty, and residents in the transition to the new assessment model.¹⁷  Review of the toolkit was mandatory for program directors and encouraged for faculty and residents. A central CBME implementation committee oversaw each program’s implementation and met with resident representatives and monitored numbers of EPAs submitted per month per program. When either residents identified concerns or EPAs submitted dropped below a threshold, members of this committee met with program leads to support adhering to the goals of CBD. At the time of recruitment, there were 120 residents enrolled in CBD at this institution.

In this institution, the acquisition of EPAs is initiated by residents. Ideally, residents identify the relevant EPA prior to the clinical encounter in order to focus the preceptor’s attention on assessment and providing high-quality feedback. Upon completion of the encounter, residents generate an electronic EPA form on a smartphone, tablet, or computer that is then completed by preceptors in real time; alternatively, the forms can be saved and emailed to preceptors for completion at a later point. A sample electronic EPA form from this institution’s core internal medicine program is shown in Figure 1. Although slight variations exist between programs, most EPA forms contain the following sections: clinical context of the patient encounter that is completed by the resident, a 5-point entrustment scale modeled after the O-SCORE (Ottawa Surgical Competency Operating Room Evaluation) from Gofton et al,¹⁸  a list of milestones as set out by our certifying body, and 2 mandatory written feedback boxes asking the assessor to highlight what the resident is doing well and what needs to improve, respectively. Independent performance by the resident is denoted numerically by a score of 5 on the entrustment scale such that the preceptor feels that they “didn’t need to be there (in theory).” An assigned score less than 5 will prompt the preceptor to evaluate individual milestones associated with the competency.

Participants were recruited through promotional posters placed in the hospital and sent to resident email listservs. For those who provided informed consent, a 30- to 45-minute interview was scheduled with the lead researcher in which participants were asked to share their motivation for obtaining EPAs, perceived importance of EPAs, and their overall thoughts on the CBME curriculum. Interviews were conducted between November 2019 and July 2020, initially in person but with the onset of the COVID-19 pandemic, subsequent interviews were conducted via Zoom. Residents received a $50 research incentive for their participation.

This is a 2-phase sequential exploratory mixed methods study, which begins with an in-depth exploration of a topic through a semi-structured interview that then supports the development of a quantitative survey to capture data from a broader group of participants.¹⁹  The qualitative and quantitative data are then integrated into a meaningful story.¹⁹  Residents could participate in only one method of data capture (eg, interview vs survey). For this study, Phase 1 captured data through a semistructured interview using a method called cognitive task analysis. A copy of the interview guide used in our study is shown in Figure 2. Cognitive task analysis is used to uncover how “experts” act within a context (eg, fraught with time pressures, complexity, uncertainty) by asking participants how they engaged in certain tasks and to reflectively structure their actions into a series of concrete and detailed steps.²⁰  In this study, residents were classified as “experts” and were asked to recall how they thought about and acquired feedback from preceptors through EPA narratives. This study aligns most with the pragmatic realist paradigm, which acknowledges the subjective experiences of participants and the nature of multiple truths associated with the phenomenon under investigation, specific to reality and one’s perception of reality.^21-24  Interviews were recorded using both an external digital recorder and the record function on Zoom. Transcription of audio recordings was facilitated by Otter.ai software (version 2.3.94-d91936ca) and screened by a research assistant for de-identification and accuracy. In Phase 2 of this study, participants were invited to complete a 2-question close-ended (with an option for open-ended elaboration on response) survey administered by SurveyMonkey. Residents were recruited through the local Postgraduate Medical Education Trainee Digest, designed to streamline communication and requests for participation. The survey captured responses to the following questions: (1) What is your primary motivation for requesting EPAs from preceptors? Response options were (a) opportunity to receive feedback about task performed; (b) mandatory program requirement; or (c) other (please specify); (2) How important do you feel EPA observations are to your learning? Response options were illustrated through a 5-point Likert scale with 1=EPAs are not at all important to my learning to 5=EPAs are extremely important to my learning. The purpose of the survey in this sequential mixed methods design was to embody the principles of equity and inclusion of diverse methods to ensure the broadest population of residents would feel comfortable contributing to the discussion. As such, the survey was designed with efficiency and anonymity in mind.

Interview transcripts were analyzed by 2 researchers: one of the investigators (D.M.H.) and a research assistant. To informally calibrate, each researcher reviewed the same 5 transcripts and engaged in open, inductive coding of the material. Once complete, the 2 researchers met to discuss their open coding and initial theme generation. The researchers continued to meet frequently to discuss their findings, negotiate content and meaning, and calibrate. In addition, interim findings were shared by the lead researcher at postgraduate committee and oversight meetings (ranging from 15 to 30 members with representation from different residency programs) to explore if and how data were situated in their broader reality. The analyses included identifying prominent themes associated with residents’ experiences acquiring EPAs as well as quantitating these findings to explore the frequency of responses through manifest content analysis (eg, quantification of identifiable aspects of text content, specifically generated themes).²⁵  Descriptive statistics captured the perspective of residents collected through survey method.

D.M.H. is an innovation and health professions education scientist. She is a mixed methodologist who uses methods to better understand complexity to uncover transferable elements of large-scale systems-level change. N.D. was a resident at the time of the study, so his viewpoint was as a learner, and V.J.D. is a clinician educator with 15 years as an attending physician. All names and identifying information were removed from transcripts prior to analysis by researchers.

This study was approved by the University of Alberta research ethics board (Pro00090407).

In total, 25 residents from a variety of medical and surgical specialties participated in the interview, and 107 residents participated in the close-ended survey with 3 residents providing written responses (see Table 1).

Program requirement was the dominant theme in resident responses regarding their primary motivation for obtaining EPAs.

• “So it’s purely because it’s a program requirement.” (Participant [P] 25)

• “The primary motivator is getting the number of EPAs I need to get to the next stage.” (P24)

• “So specifically for EPA observations, just because it’s a mandatory component, and we’re expected to do the one per week.” (P3)

Only a small subset of interviewees obtained EPAs to drive learning and self-improvement.

• “To advance my learning. Just to find my weakness, make sure that I’m actually progressing.” (P16)

• “I think the main thing is just trying to get a sense of what I should improve on to be a better physician is essentially, what it boils down to.” (P13)

• “I think most of us can get away with being okay [specialists], but I really want to make sure that my technique is the best for patients at the end of the day.” (P23)

The primary theme in this domain is that there are varying degrees of perceived importance assigned to EPA observations. On one end of the spectrum, some residents felt that EPAs were not important or only minimally important to their learning.

• “On a scale of 1 to 10, probably a 1.” (P22)

• “As in if we took [EPAs] away what would the deficit be? Yeah, it’s pretty minimal.” (P1)

• “So far, if I were to be completely frank, I don’t think [EPAs] have changed too much for me.” (P18)

In contrast, other residents greatly valued regular feedback in the form of EPAs.

• “I think they’re very, very important to my learning. I think, comparing now versus back, for example, [even in my previous specialty], or in medical school, where we did not have EPAs, I kind of just got the end of rotation eval.” (P23)

• “It actually gives you an opportunity to have very precise feedback on a specific topic and on a specific aspect of management.” (P16)

• “I think as a whole, they’re important because I like the idea of being evaluated on a bunch of different specific things.” (P2)

Three major themes emerged when residents were asked about their overall thoughts on the CBME curriculum. The first theme centered on increased resident workload and stress created by EPA requirements.

• “I mean, I feel like it’s a lot of extra administrative work for a lot of people that, you know, it makes our lives as residents more stressful I find.” (P19)

• “But it really does put all the burden on you to get all of your feedback. And it is, it does feel like a burden, especially in those tricky situations where you can’t find the right time or place to get that EPA. So, I do find that a bit stressful at times.” (P24)

The second theme among those interviewed was a concern about residents “gaming the system” to maximize EPA scores.

• “As soon as people figure out how to game the system, you can game it very well. You can pick and choose your preceptors.” (P25)

• “The thing that I am most concerned about is the cherry picking. And I would be guilty of it too, where I’d be like, well, this preceptor saw me do this, and I didn’t do great, but this one saw me do this and he was awesome and I’m going to choose this EPA.” (P4)

The third theme was an acknowledgement of lack of shared ownership by preceptors in the EPA acquisition process.

• “I think the residents are really doing their part and trying to initiate [EPAs]. But it’s the other part that has to be done which is getting the faculty on board and engaged.” (P5)

• “But now that the number requirements are really being hammered home, and it is all on us to ask. There’s no requirement on the preceptor side to initiate that process at all.” (P10)

Survey data are depicted in Table 2. Mandatory program requirement was the primary motivation for requesting EPAs for the vast majority (95 of 107, 89%) of respondents. With respect to the question about the importance of EPAs for learning, 31 of 107 (29%) residents felt that EPAs were not at all important, while more than one-third (39 of 107, 36%) of residents felt that EPAs were at least moderately important for their learning. Three residents elaborated on their responses: One reported that they are motivated to learn more about their performance on a task, while also appreciating that the task is a mandatory program requirement. One participant reiterated that their motivation is the requirement by programs. The third participant reported poor transparency by their program about the influential role of EPAs on their progression through training.

Both the interview transcripts and survey data indicate that program requirement was the primary motivator for most residents when initiating EPA assessments. Very few residents were motivated primarily by a desire to drive learning and self-improvement. Although previous authors have shown that assessment burden in CBME hampers intrinsic motivation of learners,²⁶  we are unaware of any other studies to date that have closely examined the motivations underlying resident EPA requests.

Residents in this study were divided on the importance of EPAs for their learning, with many assigning minimal to no importance compared to others assigning great value to EPA observations. This divide was mirrored by the survey results that showed similar proportions of respondents feeling that EPAs were not important versus moderately important. Resident skepticism regarding the importance of EPA observations for learning has been seen in other studies. For example, Upadhyaya et al described residents being concerned that EPAs add minimal learning value and instead represent yet another task to complete in an already busy residency schedule.⁸ 

Themes of increased resident workload and stress, “gaming the system” to maximize EPA scores, and lack of shared ownership from preceptors with respect to EPA acquisition were chief among residents’ overall impressions of the curriculum. Many of these concerns about CBME are echoed by the existing literature, including a recently published scoping review about CBME that tracked literary conversations about this training model from 1978 to 2021.¹⁶  The increased workload and stress created by EPAs has led to some residents referring to CBME as a “make work project.”⁸  Furthermore, Ott et al identified constraints on time and resources as 1 of 9 assessment burdens on residents in a competency-based curriculum.²⁶  Similarly, the admission of “gaming the system” in order to achieve a high entrustment score is not unique to the findings of this study. Multiple authors have mentioned CBME residents “cherry-picking” clinical encounters with the “goal of passing an EPA or looking good, rather than learning.”^7,8  Although other authors have identified poor faculty engagement as a barrier to EPA completion,^8,26  this study more specifically highlights resident frustration regarding the lack of shared initiative and investment by preceptors in the EPA acquisition process. The burden of responsibility for EPA acquisition has been largely placed on residents in accordance with the core CBME tenet of tailored learning experiences in which residents are to be self-directed in their learning.⁶  Unfortunately, this comes at the cost of a greater administrative burden and resultant decreased resident engagement with CBME.

Perhaps most surprising about this study was the sheer number of residents who minimized the importance of EPA observations, which we believe may be tied to their motivation for requesting EPAs. Overwhelmingly, the primary motivation for initiating EPAs was an extrinsic force: program mandated requirement. It stands to reason that for those residents who are extrinsically motivated to obtain EPAs, there is the missed opportunity to recognize the intrinsic importance of these formative learning encounters. This evokes the central tension in medical education between summative (ie, assessment of learning) and formative (ie, assessment for learning) modes of assessment.²⁷  One challenge in reconciling these assessment strategies is learner perception such that “learners may perceive assessment with formative intent as summative, restricting their engagement with it as feedback, and thus diminishing its learning value.”²⁷  This means that although EPAs were designed with a formative lens, most learners view them as summative or mandatory assessments that have questionable personal learning value.

Ultimately, the tension between CBME architects’ philosophy supporting low-stakes assessments for learning and the local on-the-ground, potentially high-stakes interpretation and application of EPAs must be reconciled in order to support the longevity of CBME.²⁸  Failing to address this disconnect may lead to inauthentic implementation of CBME in practice and subsequent activities and procedures of this training model minimized to a series of administrative rituals devoid of substance and potential impact.²⁸  Therefore, we believe this to be an urgent issue that merits the attention of medical educators and the community at large. In their discussion on learner phenotypes in CBME, Mador et al suggested faculty development, learner orientation, and education culture change as possible avenues to lessen this tension.²⁹  Others have asserted that ongoing faculty development must be the leading priority if EPAs are to be effective assessment tools long term.³⁰ 

In accordance with self-determination theory¹⁴  and competency-focused instruction,⁶  CBME as presently implemented clearly supports resident autonomy in identifying assessment opportunities. However, whether this approach supports competence by ensuring residents see the necessary mixture of cases for their growth is in question. Perhaps preceptors should have greater input on their learners’ acquisition of EPAs, an idea well supported by competency-focused instruction, which indicates greater agency among preceptors in identifying learning opportunities as opposed to being solely reactive.⁶  Based on this discussion, we propose 2 potential solutions to bridge the gap between the theoretical basis and practical implementation of EPAs. Firstly, we agree with other authors that promoting faculty engagement in the assessment process, especially as it relates to taking greater ownership of EPA acquisition, is essential, both in terms of strengthening the coach-mentee relationship and alleviating the administrative burden on learners. Secondly, we advocate for requiring a greater number of EPA observations scored at less than entrustable, thereby enabling stronger feedback and returning the focus of EPAs to their formative origins.

This study has a few noteworthy limitations. First, we operated out of a single institution where resident experience with CBD may not transfer to other institutions. Second, this study captured data during the early phases of CBD implementation, which may have influenced resident responses. Lastly, this study did not endeavor to examine the association between specific residency programs and resident attitudes toward EPAs to protect the anonymity of participants from programs with a small number of residents.

Although many residents recognize the value of EPA observations, findings from this study suggest that most of them are not intrinsically motivated to request EPAs and are primarily driven by mandatory program requirements. Concerns regarding excessive workload, “cherry picking,” and lack of faculty ownership in the EPA acquisition process potentially challenge resident engagement with CBME.

The authors would like to acknowledge Lakshmi Balaraman, PhD, University of Alberta, Undergraduate Research Funding Advisor, Student Services Undergraduate Research Initiative.

Funding: Funding for this study was provided by the Teaching and Learning Enhancement Fund, University of Alberta.

Conflict of interest: The authors declare they have no competing interests.

This work was previously presented at the International Conference on Residency Education (ICRE), October 27-29, 2022, Montreal, Quebec, Canada.