Smarter Screen Time: Integrating Clinical Dashboards Into Graduate Medical Education Open Access

Whether designing a learner-focused dashboard from scratch or adapting a prebuilt one, it is important to understand the fundamental features of a high-quality dashboard. The development and evaluation of a dashboard comprise several key steps: identification of performance metrics, consideration of data sources, creation of an effective visual display, and curriculum integration (Figure 1). The design requires collaboration with information technology (IT) professionals, administrators, and program leadership as well as input from trainees at each step of the process. In the authors' experience, programs should plan for a timeline of 6 to 12 months for the design and initial implementation phase, recognizing that the essential processes of dashboard refinement and maintenance require ongoing investment of resources. Institutional support, such as funding and IT resources, may be garnered by leveraging the need to satisfy ACGME requirements and by achieving financial goals of the health system.

Performance metrics ideally represent measures of successful patient care and population management.²⁰  Deciding on the type and number of performance metrics is paramount in dashboard design. Currently, there is no common taxonomy of metrics for different specialties; however, some reported guiding principles are to employ metrics that are resident-sensitive, educationally aligned, and appropriate to panel size and case mix.^21–24 

First, programs should identify metrics that are resident-sensitive. Resident-sensitive quality measures have been defined as ones that “require an action by the resident, with the resident possessing a realistic opportunity to do so that directly effects patient care.”²¹  For example, while lead screening is an important pediatric quality metric, if a nursing outreach initiative is driving rates of screening, resident contribution to the metric may be low and not reflect their performance. Trainee input can help generate consensus on those metrics most relevant to their practice.

Second, programs should consider how metrics align with and inform educational goals.²³  In radiology training, for example, a dashboard may display the percentage of time a resident's initial read was modified by an attending. This metric functions as a surrogate measure of their diagnostic accuracy and should improve with ongoing education. Furthermore, an analysis of reads that require substantive edits can inform resident- and program-level educational interventions.²⁵ 

Third, programs should consider what metrics are suitable given panel size and case mix. One way to address smaller and heterogeneous panel sizes is to focus on process measures, which are less subject to case mix variability. They are also more sensitive to differences in care than outcome metrics, where the measured event may be too uncommon to detect meaningful differences among residents. For example, the appropriate use of venous thromboembolism prophylaxis may be a more useful measure of surgical trainee practice than the reduction in rates of postoperative pulmonary embolism.²⁶  If used, process measures must be linked to patient-related outcomes through strong evidence.

In contrast, clinical outcomes are attractive in that they are considered the gold standard and are often readily available since health care organizations may require reporting clinical outcomes as a way to monitor quality metrics. One way to minimize the effect of small panel sizes for trainees is to choose compound metrics, like the diabetes Healthcare Effectiveness Data and Information Set. Aggregation of metrics has been shown to increase reliability of determining the physician “thumbprint.”²⁷  The concept of “the balanced scorecard” suggests the use of a range of metrics, including clinical outcome measures, process measures, and patient satisfaction scores, to provide a holistic view of the trainee.²⁸ 

Finally, it is worth considering that a dashboard not only communicates performance to learners, but also imparts the values of the program and health system.

The next step is to evaluate the data sources underpinning the dashboard. Common data sources include the electronic health record (EHR), which can yield patient data as well as process measures, patient satisfaction scores, and billing data. The choice of data source may also impact provider attribution, depending on whether data is organized at the level of the individual or care team.²⁹  For example, length of stay is a metric that better reflects interdisciplinary team function, rather than an individual's clinical management. Even those metrics that are ascribed to an individual, however, can be influenced by the actions of colleagues, preceptors, or clinical staff.²⁴  Programs could either select metrics that are more representative of independent practice or try to account for interdependence by adjusting for faculty supervision or clinic site. Emphasizing that data are a reflection of population health and opportunities rather than a report card can also alleviate potential concerns about dashboard performance and attribution.³⁰  Programs that want to use this data as summative, rather than formative, may find progress over time to be a better indicator of performance than static metrics.²⁵ 

With any metric, it is vital that programs validate data, as inaccurate or poorly attributed information will quickly undermine trust in the dashboard. EHR data validation can be done by running a similar database query and comparing results. For example, a report of patients on controlled substances could be compared with those receiving opioid prescriptions. A final validation step is to recruit a group of trainees to review results by hand for any errors or omissions.

Despite data validation, EHR-sourced dashboards are subject to certain limitations. Notably, data can only be extracted from certain fields, and free text in a note may not be recognized. Once a dashboard is rolled out, consider a reporting infrastructure to rapidly address any concerns.

Dashboards should simplify data sets to quickly convey current performance (Figure 2). To maximize impact, dashboards must have high visibility with clear infographics and images. Excessive or poorly designed features can distract learners.³¹  Color-coding data can communicate levels of performance. Designing dashboard features that are interactive and actionable may empower trainees to improve their quality metrics. For example, a dashboard element should not just show rates of breast cancer screening, but also link to a report that indicates the individual patients in need of mammographies.

Dashboard displays may show data for single or multiple providers. When using a dashboard with trainees, a program must decide whether to keep individual performance data confidential or to display information from all providers. Comparison of individual data to aggregate peer data may be helpful to benchmark individual performance within the context of the training program while maintaining privacy. Additionally, comparisons to national data can be a useful teaching tool regarding standards of care.

While the ACGME has emphasized the importance of providing trainees with practice data, there has been little research about how to meaningfully deliver this information.³²  To effectively teach dashboard skills and promote their use, programs need to make dashboards accessible and integrate them into thoughtful curricula.³³  These curricula should focus on understanding performance metrics, evaluating data accuracy, and integrating dashboards into clinical practice. Pairing education about the specific metric with the dashboard display could transform abstract medical knowledge into something tangible. To make the delivery of performance data actionable, there must be complementary quality improvement (QI) and panel management initiatives. Within GME, it is often difficult to design and implement practice-based improvement curricula due to challenges obtaining individual data, lack of faculty expertise, and asynchronous schedules.³⁴  However, there are QI tools that can be utilized with dashboards to create practice improvement curricula. For example, in internal medicine, a model for guiding practice improvement is the American Board of Internal Medicine practice improvement modules (PIMs).^35–37  PIMs are designed to guide providers or groups of practitioners, even those with limited QI training, through clinic-based improvement projects.³⁸  These types of resources allow for QI initiatives to be completed on an independent schedule. Programs can create PIMs using dashboard data and grounded in clinical quality goals and local resources.

Although direct observation of trainees for assessment provides critical information regarding behaviors, experts now suggest additional performance measures that include patient outcomes.¹⁵  A thoughtful approach to a cultural shift toward patient and population outcomes, in concordance with dashboard development, is essential, through development of a shared model among program leadership, faculty, and trainees.

Programs can empower trainees to be a driving force in the development of dashboards by including them in all steps of the process.⁷  Furthermore, trainees should understand that their medical training is linked to clinical outcomes, with dashboards acting as a potential tool for performance improvement and assessment. There is also a need for faculty development to help strengthen the connection between quality improvement, performance measures, and medical education. Financial linkage of clinical outcomes and medical education can also help drive and expedite cultural change.⁶ 

Limitations to the use of dashboards in medical education include the potential inability to attribute the contribution of a single trainee to a performance measure, difficulty comparing performance measures between different clinical settings, and insufficient data points to render an adequate performance evaluation.⁶  While the EHR is the most obvious choice for dashboard placement, many are not optimized to support dashboard integration due to issues with functionality, display, and inadequate data mining ability.⁸  Furthermore, dashboard implementation requires a large amount of monetary and health IT resources.

From an academic standpoint, further research is necessary in evaluating the impact of dashboards within GME, delineating financial and personal costs for dashboard creation, and identifying the optimal way to integrate dashboards within educational curricula. We also suggest that governing boards such as the ACGME increase their support for the use of clinical performance data in training.

With the growing use of big data analytics, the aggregation and display of personalized metrics will play an increasing role in medicine.³⁹  Electronic dashboards show promise for improving the clinical care of patients as well as medical training. To expand their use in GME, dashboards should include valid data and be designed such that they are accessible, actionable, and clinically relevant. Electronic dashboards that can meaningfully distill complex data into useful information may promote advancements in preventive medicine, population health, and systems management. In GME, a cultural change is needed to support the integration of clinical performance data through dashboards.