Does YouTube offer high-quality ınformation? Evaluation of patient experience videos after orthognathic surgery

Although the esthetic awareness and expectations of people, especially regarding their facial appearance, have increased, demand and interest in orthognathic surgery has also increased. The aim of orthognathic surgery is to increase patient satisfaction by improving a patient's facial esthetics, psychological state, and jaw function.^1,2  In previous studies, the following factors were shown to play a role in patient satisfaction or dissatisfaction after treatment: age and gender, duration of treatment, expectations from treatment, providing adequate information to patients, explaining the surgical procedure and complications, patient-physician communication, psychosocial status before and after surgery, and the influence of the individual's environment.³ 

It was observed that a more positive approach toward the surgical procedure and its results could be obtained and fewer problems would be experienced by better informing and educating the patient before surgery.⁴  In their study, Olson and Laskin stated that, if the situations that may be encountered during the operation were explained in detail and all kinds of information were given, the psychological trauma of patients may decrease and their satisfaction level may increase.⁵  Though surgical repositioning of the jaws is safe, complications can occur as with any surgical procedure. It is important that patients are adequately informed about potential risks before surgery.^6,7  In general, it can be said that the negative effects seen at the end of the operation are not sufficiently understood by patients.⁸ 

In addition to being used more for communication purposes in recent years, social media and the internet are also a comprehensive source of information that is frequently used in the field of health.⁹  Patients generally prefer to use YouTube, one of the most popular websites, because of its visual and auditory information compared to other social media platforms.¹⁰  Particularly for orthognathic surgery patients, YouTube shares by people who have gone through this process can affect the treatment positively or negatively depending on the quality of the content of the information accessed. The fact that users do not have sufficient resources and competence to evaluate the quality and validity of this information requires the evaluation of YouTube videos with certain objective scales.¹¹  Therefore, the aim of this study was to evaluate the content, reliability, and quality of videos of patient experiences after orthognathic surgery on YouTube.

Ethics committee approval was not required for this study as it contained only publicly available data. At the beginning of the study, three terms (orthognathic surgery, double jaw, jaw surgery) frequently used by individuals seeking information about patients who had undergone orthognathic surgery were determined using the Google Trends (Google Trends, 2020, Alphabet, USA) website. “Keywords” and “sort by relevance” options were used as default filters. A total of 50 videos that met the inclusion criteria were analyzed (Figure 1). Videos watched on the same day on the YouTube website (https://www.youtube.com) (May 2, 2022) were evaluated separately by two researchers. The informative quality of the published videos was evaluated. Since there were a limited number of videos, no distinction was made between English and Turkish videos, and there was no restriction on the length of the video. All videos were fully viewed and a few general parameters were recorded for each video: number of views, number of comments, number of likes and dislikes, length of video in seconds, and number of days after upload. In addition to these, viewing rate and interaction index were also calculated (Figure 2).

A 13-point scale was used to evaluate the content of the videos: (1) description of orthognathic surgery, (2) preoperative treatment procedures, (3) orthognathic surgery procedure, (4) comparison of treatment options (camouflage therapy or early functional and orthopedic treatments), (5) postoperative edema and paresthesia, (6) pain, (7) oral hygiene, (8) soft tissue pain, (9) speech performance, (10) psychosocial aspects, (11) treatment cost, (12) treatment success, and (13) duration of treatment. For each piece of content, the videos were rated 0 (not included) or 1 (included), and the total content score was determined. Those with exactly 7 points or less than 7 points were classified as poor-content videos and those greater than 7 points were classified as rich-content videos. The total content score was obtained by summing after each score was awarded (Figure 3).

The videos were evaluated with the Video Information and Quality Index (VIQI) and the Global Quality Scale (GQS). VIQI includes four rating criteria. For each criterion, videos are scored on a 5-point Likert scale: information flow (VIQI 1), information clarity (VIQI 2), video quality (VIQI 3), and consistency (match between the title and content of the video) (VIQI 4).¹²  For the GQS assessment, videos were scored between 1 (low quality) and 5 (high quality) (Figure 2).¹³  To assess the reliability of the information, the DISCERN (written quality criteria for consumer health information about treatment options) scoring system was used.¹²  DISCERN consists of three parts and 16 questions. The first part evaluates the reliability and consists of eight questions; the second part includes seven questions that evaluate the quality of information about treatment options. In the third part, the quality of the website is determined with one question. Scoring is done in the range of 1–5 for each question. A “5” indicates that it meets the criteria, and “1” indicates that no criteria are met. Websites are divided into five groups: 16–26 points are “very low,” 27–38 points are “low,” 39–50 points are “moderate,” 51–62 points are “good,” and 63–80 points are “very good.”¹⁴  A reliability score was obtained by calculating and evaluating the sum of these scores.

According to the inclusion criteria, evaluation and scoring of all videos selected were completed by two calibrated observers. A third observer was consulted in case of any inconsistency. Statistical analyses were performed in the SPSS software program (version 21.0 SPSS Inc, Chicago, Ill). The normality of the data was evaluated with the Kolmogorov–Smirnov test, and nonparametric statistical analyses were performed since all the parameters used in the study were not normally distributed. The number of views of each video, number of likes and dislikes, number of comments, video duration, time elapsed since the day it was uploaded, viewing rates, and interaction indexes were calculated. Descriptive statistics of the parameters are shown as median, mean, standard deviation, minimum, and maximum.

All parameters used in the study were divided into two groups as poor and rich content according to the total content score. Comparison was made using the Mann-Whitney U-test. Also, the possible correlations between the total content score, total VIQI score, GQS, DISCERN scores, and duration, like, dislike, comment, and viewing rate data of the videos were analyzed by calculating the Spearman correlation coefficients. Additionaly, linear regression analysis was used to determine the relationship between GQS, total VIQI score, DISCERN, duration, and total content scores. All videos were re-evaluated 2 weeks later by the same researchers (E. A. and M. H. B) to calculate within-observer reliability. Intraclass correlation coefficients (ICCs) were calculated to determine intra- and inter-observer reliability. Statistical significance was P < .05.

Descriptive statistical data for the included videos are shown in Table 1. The average number of views was 57,640.02, the average number of likes was 712.78, the average number of dislikes was 31.08, the average number of comments was 187.54, the average time (days) from the time the videos were posted was 797.26, and the average length of the videos (seconds) was 645.88. The average interaction index was 1.23 and the average viewing rate was 124,831.67. On average, the content score was 6.94, the GQS score was 2.58, the DISCERN score was 2.66, and the total VIQI score was 10.58. In the intra- and inter-observer evaluations for measurement reliability, the repeatability coefficients for all parameters were found to be acceptable (0.891–0.948 and 0.916–0.977, respectively).

The most frequently mentioned postoperative complaints in the YouTube videos included in the current study were pain (34%) and swelling (34%). Although 16% of the patients mentioned a general pain complaint that they could not describe, 12% explained it as a headache. Decreasingly, 6% of patients complained of numbness and 4% of patients complained of nausea. Besides these, 22% of the patients did not mention any discomfort (Figure 4).

When videos were considered as a source of information, the variables between rich- and poor-content videos are compared in Table 2. It was determined that the number of likes (P < .05), the number of comments (P < .05), video durations (P < .001), and interaction index (P < .01) were greater in high-quality videos than low-quality videos. However, there was no statistically remarkable difference between the number of views of the videos, number of dislikes, time elapsed since they were published, and rate of watching between the videos with rich and poor content (P > .05). Compared to the group with poor content, the rich-content group also had higher total content score, GQS score, DISCERN score, and total VIQI scores (P < .001).

Possible correlations between total content score; total VIQI score; GQS; DISCERN scores; and the data of duration, like, dislike, comment, and viewing rate of the videos are shown in Table 3. Except for the age of the video, there was a statistically significant correlation between the characteristics of the videos and the scoring of the videos in terms of quality and content (P < .05). Total content score, total VIQI score, GQS, and DISCERN scores were also found to have a statistically remarkable positive correlation among themselves (P < .05). In linear regression analysis, the relationships between total content score and time (r² : 0.44, P < .001), GQS (r²: 0.47, P < .001), DISCERN (r²: 0.46, P < .001), and total VIQI score (r² : 0.53, P < .001) were statistically significant.

It has been stated that patients who have more complaints about symptoms such as pain, swelling, and numbness after surgery lack the information stage before surgery. In a previous study, it was observed that individuals who were aware of these results experienced fewer problems after surgery.¹⁵ 

Preoperative knowledge of orthognathic surgery patients and cooperation with the physician are very important in the potential success of treatment. Many of the patients researching this treatment have easier access to social media, unlike written information on scientific platforms.¹⁶  Considering the recent increase in the use of the internet and social media, physicians should be aware of the content and reliability of the information shared so that they can guide patients appropriately.¹⁷ 

It is inevitable that it would be the right approach to think that most of the patients who will undergo orthognathic surgery will seek information about their treatment processes through the YouTube video sharing platform.¹⁸  However, besides the fact that it contains a lot of educational videos, considering the ease of video sharing and recent developments and changes in some of the information in the videos, it may be suspected that it may be out of date or inaccurate.¹⁹ 

According to the content analysis in the current study, the video quality of the patient experience after orthognathic surgery was found to be at an average level. Although most videos talked about complications such as the operation process, pain and swelling, they were not fully consistent with each other, and most of them were about praising the treating physician. The number of videos that visually supported the process and touched on issues such as duration of pain, its severity, and cost of treatment, was very limited. Considering these scenarios, it can be said that YouTube videos were moderately sufficient in information. In addition to specialized physicians, patients can upload videos to YouTube independently of them; that is, there is no standard created and controlled by the system.^20,21  In a previous study evaluating YouTube videos related to orthodontics in accordance with the current findings, it was reported that the quality of the videos was at an average level and their reliability was low.²² 

It was observed that videos with rich content contained much more informative explanation than videos with poor content, and accurate and balanced information is more beneficial for patients. In the current study, the rich-content video groups had higher DISCERN scores and those were closely correlated with the GQS score. A moderate correlation was found with the content score. It can be accepted that the accuracy and information quality of the videos with higher DISCERN scores were relatively high.²³  It can be said that completing the information flow with many visual informative elements in the videos increased the quality and content score. The criteria of video flow, information accuracy, quality, video title, and clarity were found to be higher in the rich-content video group than in the poor-content video group. Videos with rich content had a higher total VIQI score than videos with poor content.²⁴  In contrast, in another study, no difference was found in the total VIQI score between the poor- and rich-content video groups. This can be explained as a result of YouTube users following the developing technology.

The determination of content titles for the evaluation of videos can directly affect the analysis of video quality and its accuracy. In the current study, for videos with rich content, the information content, number of views, number of comments, duration of the video, and interaction index were higher than the videos with low-quality information content. It is an important criterion when evaluating videos, as more detailed information can be given as the duration of the videos increases. Explaining the subject in detail can give the correct information completely.^25,26  However, it should be taken into account that prolonging the time may also cause loss of concentration.²⁷  To inform in the literature, the length of a shared video should be determined according to its target; It has been said that the shorter the video, the greater its impact on the subject may be.¹⁶ 

Although the time elapsed since the upload date was found to be statistically insignificant, it was numerically higher in videos with poor content. In studies in the literature, it was reported that the interaction index and video duration and video information content quality showed a positive relationship with each other, and that the interaction index and video duration will increase as the information content increases.¹⁹  There are a limited number of videos about the patient experience after orthognathic surgical treatment of YouTube users. It can be said that the patient watches at high rates without discrimination; and can interact with other users through the features of comments, likes, and dislikes; but because a small number of videos are uploaded, the opportunity continues to be lacking for patients to obtain information or make comparisons.

The rate of watching videos and the number of likes and dislikes of YouTube content are constantly changing. New videos can be added every day, old videos can be deleted, and the lack of standardization for videos should not be overlooked in the evaluations. Despite these limitations, it is thought that this method can be considered reliable for patients who want to get preliminary information. In the current study, it was thought that the information content of YouTube videos about the patient experience after orthognathic surgery was missing. Therefore, it is necessary for professional physicians to guide patients who want to have surgery correctly regarding the reliability of the information contained on this platform.

• YouTube videos in which patients share their own experiences after surgery are generally of medium quality. Rich-content videos contain much more informative explanations, video accuracy and quality are higher, and balanced information is more beneficial for patients than poor-content videos.

• Within the limitations of this study, YouTube videos in which patients who have undergone orthognathic surgery share their experiences can be considered as a source of motivational information for patients who want to undergo surgery. However, it should be ensured that physicians who are specialized in this field question the quality of video content. In addition, videos produced by associations such as the American Association of Orthodontists or the American Association of Oral or Maxillofacial Surgeons can be a guide for patients and physicians, with higher quality videos that can be easily used by clinicians and easily accessed by patients.