Next Steps in the Implementation of Learning Analytics in Medical Education: Consensus From an International Cohort of Medical Educators

The availability of increasing amounts of data about learners and their performance presents new challenges and opportunities in medical education around the world.¹  In North America, the transition toward competency-based medical education (CBME) by the Accreditation Council for Graduate Medical Education (ACGME) Milestones and the Royal College of Physicians and Surgeons of Canada's Competence by Design is leading to an increase in the amount and diversity of trainee performance data.^2–5  A programmatic approach to assessment in CBME supports the use of this assessment data of, and for, learning.⁶  These data have the potential to provide a more holistic view of each learner's progress than traditional assessments, facilitate individualized teaching, coaching, and assessment, inform remediation planning, predict future performance, establish learning trajectories for various competencies, contribute to faculty development and program evaluation, and potentially tie educational assessments to patient care outcomes.⁷  However, sophisticated techniques to analyze and display data for these purposes have not been widely applied in medical education to date.⁸ 

The field of analytics involves the collection and analysis of data, often through statistical modeling, to develop actionable insights.⁹  Analytics support a variety of decision-making activities across many fields, including business and sports.^8,10  The use of learning analytics, a subtype of analytics that interprets educational data to describe, characterize, support, and predict the behaviors of learners in higher education, has recently proliferated.^8,10  A lack of awareness and application of learning analytics methodologies have been cited as barriers preventing the widespread use of these techniques in medical education.⁷ 

We sought to characterize barriers to the use of learning analytics techniques in medical education by identifying the questions of educators interested in this field. We anticipate that the elucidation of these questions will better characterize current gaps in knowledge and policy that need to be addressed to potentiate the effective use of learning analytics.

The Summit on Learning Analytics in Medical Education (the Summit), hosted during the 2017 Royal College of Physicians and Surgeons of Canada's International Conference on Residency Education (ICRE), served as a venue to explore this study question with an international sample of interested educators.

Attendees of the Summit at ICRE were recruited to participate. The Summit was a 2-day (October 18–19, 2017) preconference event with required preregistration, organized by the authors of this article and supported by the Royal College of Physicians and Surgeons of Canada. The target audience was medical educators and medical education scholars with an interest in learning analytics in medical education. Workshop sessions were presented on the use of learning analytics at various learner and program levels, potential sources of data for learning analytics, challenges in using learning analytics to facilitate learning, promises and pitfalls in the use of learning analytics, the use of clinical outcomes for assessing learning performance, the use of dashboards to provide formative and summative feedback, privacy and data legacy issues, and key questions in learning analytics.

Data collection occurred in the afternoon of the second day of the Summit during a 1-hour session focused on key questions in learning analytics that happened just prior to the summary and closing events. The research purpose of this session was clearly explained to the participants, and consent was presumed based on their voluntary participation.

The operational definition of learning analytics was defined at the Summit as the interpretation of educational data to describe, characterize, support, and predict the behaviors of learners.^8,10  Prior to the Summit, we identified 3 broad dimensions related to learning analytics (learner perspectives, program perspectives, and data stewardship) through a review of key literature^7,8,11,12  and discussion. At the beginning of the Summit session, one author (B.T.) described each dimension to all participants using the discussion prompts outlined in table 1. We then requested that the Summit participants divide themselves into 3 groups of relatively equal size to discuss these topics in relation to learning analytics. Paper copies of the topic statement and guiding questions were available to each group.

Each session lasted approximately 15 minutes and was repeated a second time (6 discussions in total). Participants self-selected which 2 of the 3 discussion groups to attend based on their interests. The goals of the small group sessions were to (1) facilitate conversation surrounding key questions related to the topic and the use of learning analytics in medical education, and (2) capture these key questions for further analysis. Three discussion groups were facilitated by faculty with expertise on the topics: learner perspectives (K.E.H.), program perspectives (S.H.), and data stewardship (B.T.).

The format of the small group discussions was designed to optimize the number of questions submitted and was based on the first 4 steps of the nominal group technique (introduction, silent generation of ideas, sharing ideas, and group discussion).^13,14  To ensure that participants' initial thoughts were captured, the facilitators instructed them to submit key questions related to their topic as soon as they arrived in the room. Participants then engaged in small group discussion until they were paused by their facilitator and given time to submit additional questions. Participants submitted questions anonymously from their devices using Poll Everywhere (San Francisco, CA) audience response software. They could choose not to participate in the question generation exercise, and submissions were not tracked by the submitter.

This research received an exemption from the Institutional Review Board of the American Institutes for Research.

Following the Summit, we collated the submitted participant questions on a Microsoft Excel spreadsheet and conducted an inductive content analysis.¹⁵  When multiple participant questions were entered as part of a single submission, we separated them into individual questions.

Three investigators (B.T., E.W., K.E.H.) independently reviewed all questions, generated potential codes, and met 4 times over 5 months to discuss and combine key ideas into a codebook through consensus. Once the codebook framework was established, one author (B.T.) recoded all data while 2 others (E.W., K.E.H.) each coded half of the data independently. The 3 reviewers then met again to discuss discrepancies; when possible, these discrepancies were resolved through consensus of the disagreeing raters. When there was disagreement between the 2 coding investigators, the third (E.W. or K.E.H.) adjudicated. When necessary, questions were classified under 2 codes. These 3 investigators reviewed the coded data to group the findings into larger themes and subthemes. Finally, one author (B.T.) drafted representative questions within each subtheme with the intention of accurately representing the essential constructs contained within the raw data. These questions were reviewed and revised by the 2 other investigators (E.W. and K.E.H.) to ensure that no important ideas were missed. The full authorship team, all of whom participated in the Summit, then reviewed results for clarity, cohesiveness, and completeness.

The 3 reviewers considered reflexivity in their work. They included the program director of a large internal medicine residency program with a background in quality improvement (E.W.), the dean of assessment of a medical school (K.E.H.), and a junior faculty member with training in medical education (B.T.). Throughout the coding process, they challenged one another regarding the applicability of the questions from the dataset in different settings and discussed their own experiences and perspectives.

A total of 67 participants (39 male, 28 female) attended the ICRE Learning Analytics Summit from 6 countries, including Canada (43), the United States (14), and Australia/Europe (10). The primary affiliation was listed as an academic institution (eg, a university) for 39, a national medical organization (eg, Royal College of Physicians and Surgeons of Canada) for 17, a clinical institution (eg, a hospital) for 7, and a corporation for one. Three participants did not provide an affiliation. Most of the Summit participants attended the “Key Questions in Learning Analytics” session, but exact attendance was not recorded.

During the 6 small group sessions, participants submitted 195 questions. Three questions were deleted because they lacked sufficient information to code. The final dataset comprised 192 participant questions: 72 from the learner perspectives groups, 76 from the program perspectives groups, and 44 from the data stewardship groups.

Table 2 outlines the themes resulting from the content analysis with representative questions listed under each theme. The analysis revealed 3 main themes (implementation challenges, data, and outcomes) and 18 subthemes. During coding, 25 items received 2 codes, and 9 items were coded as a theme without a subtheme: 1 as implementation challenges, 3 as data, and 5 as outcomes. Minor modifications to the themes, subthemes, and representative questions were a result of feedback from the authorship team, but no new questions or codes were added following their review.

This theme was featured prominently in the analysis, suggesting that implementation challenges may be as much of a barrier to the use of learning analytics as their technical challenges. The subthemes indicate that the use of learning analytics is also a complex change management problem. Participants raised the issues of gaining buy-in, building trust, providing support, and overcoming resistance.

This theme raised questions regarding all aspects of learner data, including its collection, security, governance, and access. Subthemes addressed the analytic techniques used, how the data should be presented to various parties, and the validity of evidence for using learning analytics to make decisions regarding trainees. These challenges highlight the technical capabilities and data policies that must be addressed at individual sites.

This theme outlined issues regarding how learning analytics could quantify the outcomes of learners and faculty as well as the evaluation of programs and systems. Subthemes regarding the purpose of learning analytics and its potential effects on educational and clinical outcomes suggest that work is needed to better define the objectives of learning analytics, which may help inform why they are, or are not, being used.

This qualitative analysis of data collected from an international cohort of medical educators identified 3 major themes pertaining to the application of learning analytics in medical education. The scope of the subthemes and numerous representative questions underscores the confusion regarding learning analytics that existed in an interested group of stakeholders, even at the end of a Summit on this topic. Addressing these issues is likely necessary for learning analytics to be appropriately and effectively implemented in medical education.

Across the 3 identified themes, there was a strong focus on needing to better understand how learning analytics might impact learning and learners. Within higher education, a learning analytics cycle has been described that underscores the importance of “closing the loop” with learners and ensuring that the data collected about them is fed back to them.¹⁶  This process presents data as actionable analytics for the purpose of more rapid and efficient educational intervention.¹⁷  It could include displaying their learning data (relative to peers, historical cohorts, or predetermined performance standards), having faculty initiate personal contact when these data indicate that learners are struggling, or even demonstrating how the information collected resulted in changes for future cohorts.^16,18  Subthemes in this study that related to how learning analytics will change assessment are similar to calls for transparency in the use of assessment data appearing in the higher education literature.^12,16  Overall, our results suggest that further study of the impact of learning analytics on medical learners is needed and underscore the need for learner involvement in the local development and adoption of learning analytics initiatives.

Implementation challenges were identified as a primary theme. Beyond the need for supporting learners and faculty to adapt to an environment in which more data are generated about and for them, the change process itself was a concern. This recognition is prescient because the effective use of learning analytics can alter how data are used and analyzed by the organization, faculty members, learners, and programs in undetermined ways. For implementation to be successful, significant work must be undertaken to engage the stakeholders, communicate effectively, provide accessible ways to use the data, and integrate the implementation in ways that are closely tied to institutional priorities.¹⁹ 

After data collection begins, it will be important to determine how change management processes and implementation plans can be used to optimize the use of learning analytics. Within the education literature, the RAPID Outcome Mapping Approach (ROMA) is a promising example.¹⁹  Developed for complex institutional contexts such as health care, ROMA serves as a holistic 7-step framework to understand and develop strategies for a challenge like learning analytics implementation.¹⁹  Given the prominence of implementation challenges within our analysis, local use of learning analytics will likely benefit from consideration of the literature addressing leadership and implementation science to overcome these barriers.^20–22 

Themes concerning the security and governance of learner data were more numerous than those related to its analysis, highlighting the prominence of these concerns within this group of stakeholders.⁷  While the digitization of information has made learning analytics possible, it also introduces the possibility of data breaches, which have the potential to expose data and create significant consequences for learners.^12,23  Close attention to security and governance is essential to ensure that key stakeholders trust the system with their data.¹²  Fortunately, these challenges have already been addressed in other fields. The Society for Learning Analytics Research describes various analysis techniques that protect learner data.¹¹  Best practices for the maintenance and protection of patient data have been published and could inform the stewardship of learner data.^24,25  An important question within the medical education sphere is who should have access to various types of learner data. Program directors, rotation supervisors, competence committees, academic advisers, and other educational leaders may require access to some aspects of learner data, but local policies are needed to determine who should have access to what type of data, when, and for what purpose. Clearly and transparently addressing security and governance concerns is likely to facilitate the implementation of learning analytics at the local level.

The outcomes theme focused on assessment and evaluation and suggested that learning analytics can facilitate programmatic assessment and support learners.^16,26  Whereas traditional assessment occurred within discrete courses or clinical rotations, learning analytics make it possible to perform sophisticated analyses of a learner's developmental trajectory over time and across programs.^27–29  In addition, programs can use learning analytics data predictively to identify risks for poor performance and intervene to change a learner's trajectory.^7,30,31  Learner assessment and program evaluation in medical education draw on models and theories that are well described in the literature.^32–34  However, the use of more sophisticated techniques for the analysis and visualization of learner data for assessment and evaluation are still relatively novel^11,18,35  and can support the use of these theories in practice in a targeted way. From the individual learner perspective, learning analytics offers insights that contrast effective and ineffective learning behaviors.¹⁰  The opportunity to consider learning analytics in the context of learning sciences literature suggests opportunities to promote adaptive approaches to learning that support conceptual understanding and long-term retention.³⁶  With local learning analytics implementation, it will be important to clearly articulate the purpose and specific outcomes desired while also studying their implementation to demonstrate that these goals are being achieved without unanticipated negative consequences.

This study has limitations. Participants were a self-selected group, and were more likely to be engaged and interested in learning analytics than other medical educators. While this is advantageous in identifying important questions, some important subgroups such as learners may have different perspectives and were underrepresented in the group. We cannot confirm how many of the Summit participants submitted questions or how many questions were submitted by each attendee. Although the question submission process was anonymous, it is possible that participants had additional questions that they chose not to submit. More questions may have been submitted had all participants been able to attend all 3 sessions; however, time was not available for a third small group discussion. The positioning of the session at the end of the Summit ensured that the participants had recent exposure to current issues in medical education learning analytics, but the information presented could have influenced their perspectives on the topic. Data were collected in October 2017; thus, it is possible that educators' sentiments have changed. Finally, we did not have a mechanism to prioritize the questions or verify our results with the participants following the conference.

Our analysis determined that issues related to implementation, data management, and outcomes may limit the adoption of learning analytics in medical education. These results may provide educators with a framework to address these critical issues broadly and in their own context through stakeholder education, research, and policy development.