Design and Implementation of a Peer-Reviewed Medical Education Video Competition: The Best of American Thoracic Society Video Lecture Series

Video is increasingly the medium of choice for people consuming content online, and crowd-sourced video sites such as YouTube serve as a primary driver of growth in video consumption.^1,2  Limited data regarding video in medical education demonstrate that video is effective for acquiring knowledge^3–5  and developing technical skills.^6–9 

Despite video's increasing popularity as a medium of instruction, a significant drawback is a lack of peer review to ensure accuracy. Prior studies have confirmed substantive variation in quality and accuracy of crowd-sourced medical education videos and in contributing authors' credentials.^10,11 

Among pulmonary, critical care, and sleep medicine (PCCSM) clinicians, the American Thoracic Society (ATS) is a trusted resource through the society's journals and online platform. We describe the development of the Best of ATS Video Lecture Series (BAVLS), the first professional society-based, open-access site dedicated to crowd-sourcing peer-reviewed educational videos for medical learners and health care providers.

A review form to evaluate submissions was created based on consensus input from the BAVLS steering committee (figure 1a). A plan-do-study-act (PDSA) process was utilized to develop and iteratively modify how videos were reviewed (figure 1b).¹²  The number of reviewers used per submission, assessment criteria, scoring scales, and invitations to revise and resubmit evolved through careful review and discussion. Video reviewers were experts in medical education, educational research, and scientific investigation.

During the first 2 cycles, 2 reviewers per video independently assessed and submitted scores. Over following cycles, using a PDSA approach, it was determined that expanding the ranges and providing descriptive, contextual anchors would improve accuracy and reliability of reviewers' scores. To provide feedback that may lead to better quality videos, “accepted pending minor revisions” and “not accepted but option for revision” categories were added (figure 1a, cycle 3).

During submission cycle 1, there were 7 review criteria (figure 1a), with an emphasis on brief (<15 minutes) and accurate content. This was consistent with best practices for educational videos, since learner engagement decreases as video length increases, and learners are more likely to remain attentive and retain material from shorter videos.^13,14  Some assessment criteria were revised over several cycles to optimize interrater reliability. For example, consensus across reviewers was reached that the “evidence-based content” criterion in cycle 1 did not contribute to defining a video as effective and to engaging in covering the topic of interest, and that the “accuracy” criterion was sufficient in detecting inaccurate content. Thus, this criterion was removed. Three additional criteria were added for cycle 2 in an effort to optimize interrater reliability and to identify videos that would be of most interest to the target audience. The new criteria included: (1) Relevance to the ATS mission (video topic must align with at least 1 of 3 pillars of the ATS: critical care, pulmonary, and sleep medicine); (2) Organization (flow of the video is smooth and easy to follow); and (3) Subjective (video is engaging and watchable). The current review form reflects the final review criteria (provided as online supplemental material).

To develop a broad content base, several approaches were taken to promote video submissions from the PCCSM community. First, the value of prominent publication of submitted videos on the BAVLS website was emphasized in requests for submissions, as the reputational benefit of having one's video hosted on ATS platforms has academic value. Second, peer review and the competition format promoted a focus on quality. Third, cash prizes ($500, $250, and $100) were provided each cycle to the 3 submissions determined by expert review to be the best videos.

All videos are open access. Since learners use a variety of platforms to access content, accepted videos are posted to both the society website (www.thoracic.org/professionals/clinical-resources/video-lecture-series/index.php) and an ATS YouTube channel (http://bit.ly/ATS-video). Beginning in 2016, accepted videos were posted to coincide with the ATS International Conference, where top videos and their creators were publicly recognized.

Success of the BAVLS platform was evaluated by conventional benchmarks of web engagement, including number of views, duration of views, and time on site, which were tracked continuously since implementation of BAVLS in 2016. In addition, viewers' country of origin was tracked to assess the global breadth of BAVLS videos.

The numbers of video submissions increased between 2016 and 2017 (figure 2a). The sustained interest and rate of submissions in the second year of BAVLS enabled the transition to a year-round program in late 2017, with both spring and fall submission deadlines.

By April 2017, total views of BAVLS videos were in the thousands on both the ATS website and YouTube (figure 2b), and views per video were 479 to 921 across sites (figure 2c). There was a rapid rise of 867% in total views in the last year (figure 2b). Mean video duration was 9 minutes 15 seconds, and time on site for the 75 videos viewed more than 500 times on either portal was 3 minutes 40 seconds, 442% higher than the mean of 50 seconds for all education, research, and advocacy content across the ATS website. The mean video retention rate (percentage length of video watched) was 40%. BAVLS has also demonstrated global reach with views from 89 countries on 6 continents.

The percentage of accepted videos decreased when the number of reviewers per video increased in cycle 3 (figure 2a). More errors were found and more requests for revisions were made. There were temporal improvements in innovation and quality of the videos submitted to BAVLS as well. In the first cycle, 26% of videos accepted (5 of 19) were screen-capture talks that were effective but did not utilize modern pedagogy. By cycle 4, only 7% of accepted videos were screen-capture. Over time, innovative video techniques have been employed more frequently, such as advanced postproduction editing with chroma keying techniques, and the use of scannable QR codes within a video linking to a calculator, allowing interactive determination of transpulmonary pressure.

We describe a professional society–based program to solicit, review, and host medical education videos. This has enabled a diverse group of medical educators from varied academic settings and areas of expertise to collaborate on the review process. Expert review to carefully scrutinize videos for errors and problems may have increased viewers' confidence in the accuracy of videos, as large increases in views were seen over time.

The review process emphasizes creativity and innovation in video-based pedagogy, which resulted in selecting videos that feature multiple and varied educational approaches. This collection provides diverse content while catering to different learning styles. Working in partnership with an international subspecialty society allowed for submissions from around the world, which enriched educational content on the platform. Societies are uniquely positioned to implement similar programs due to their global reach for solicitation and dissemination of videos, as well as access to a diverse group of both content and medical education experts for peer review.

The greatest strength of the BAVLS program has been the standardized and consistent review process for submitted videos. Using PDSA methodology to iteratively assess and modify the review process improved the perceived accuracy and reliability of reviews, which is critical to submitters as well as viewers. Increasing the number of reviewers per video led to higher detection of errors and more requests for edits to the video, which may have increased the quality of videos accepted and posted.

This program is limited by the focus on improving the quality of videos without a strategy to determine if the videos changed viewer learning or behaviors. Adjustments to the review process may have led to higher-quality videos with greater use of innovative techniques submitted in later cycles. However, while innovative technologies on content delivery have not been studied, multimedia design theory posits that providing coherent visual and auditory content may be more important than innovation for the sake of innovation. Instead of using Delphi methodology, we used a practice dataset prior to implementing review criteria, which may have delayed the use of more stringent criteria and scoring methods in video reviews.

As this video collection grows, future research should evaluate the relationship between video length and retention rate since the largest study of educational videos to date has shown a sharp drop off in retention after 6 minutes.¹³  It would also be useful to assess whether innovative technologies add to viewer engagement and knowledge retention.

The development of the Best of ATS Video Lecture Series has led to the creation and posting of many high-quality peer-reviewed videos via open access platforms. These videos have had large increases in views over just a couple years, suggesting a high level of demand by viewers.