Evidence-based orthodontics: Current statistical trends in published articles in one journal

The 21st century ushered in the age of evidence in dentistry.1 Evidence-based dentistry (EBD) is “an approach to oral health care that requires the judicious integration of systemic assessments of clinically relevant scientific evidence relating to the patients' oral and medical condition and history, with the dentist's clinical expertise and patient's treatment need and preferences.”2 The purpose of EBD is to “close the gap between what is known and what is practiced.”1 Peck stated: “We have an obligation to be clinical scientists providing the best evidence-based service.”3 

The EBD model paradigm for clinical decision-making has three hierarchical levels. From a pragmatic view, only levels 2 and 3 are significant and truly evidence based.4 Level 1 is the lowest level and is based on information derived through a practitioner's clinical experience. Level 2 is information obtained from the clinician's experience but, more importantly, knowledge acquired from a review of selected published articles and research. The literature review in Level 2 is not exhaustive and could be limited to a cursory evaluation of several articles that are readily available to the practitioner. The highest level of evidence is Level 3, which consists of a systematic review that attempts to search all the literature and typically culminates in the use of meta-analysis. A goal of the Level 3 literature search is to select the best studies from all those that were reviewed. The best studies are typically randomized controlled trials (RCTs).5 The merit of systemic reviews and RCTs have been well documented.2,6,7 To facilitate evidence-based decision making, guidelines such as CONSORT (Consolidation Standards of Reporting Trials), QUOROM (Quality of Reporting of Meta-analyses), and MOOSE (Meta-analysis of Observational Studies in epidemiology) were established.7,8 

There will never be enough Level 3 evidence to address the myriad of issues in orthodontics; therefore, practitioners will often be compelled to rely on EBD Level 2 evidence. Level 2 obligates the orthodontist to have an understanding of the principles of science and evidence, including research design and statistical analyses. Proffit9 wrote: “The orthodontic practitioner is akin to the scientist who must continually evaluate new research findings.” Certainly, an understanding of statistics is a necessary requisite to the evaluation of published clinical studies. Brandt and Gianelly10 stated: “Statistical analysis is at least as important for the clinician as it is for the researcher, since he has to evaluate both clinical and research materials to improve his understanding and skills in treating patients.” An evaluation of an investigation's statistical analyses can provide much useful information about the study's research design.

It would be important for orthodontists to know which statistical tests are being used in orthodontic publications and whether there were any time-related trends. Rinchuse and Zullo11 and Rinchuse et al.4 investigated the types of statistics used and made comparisons among American Journal of Orthodontics and Dentofacial Orthopedics (AJODO) articles published in 1975, 1985, and 2003. There has been a progressive increase in the complexity of research designs and the use of inferential versus descriptive statistics in AJODO articles in the past quarter century. There was also a dramatic increase in the percentage of articles using inferential nonparametric versus parametric procedures from 1975 to 2003. In 1975, 43.1% of original articles used statistical procedures, whereas in 1985 and 2003, 75.9% and 94% of original articles, respectively, used statistical tests. The AJODO published 72 original articles in 1975 and 87 articles in 1985. The percentage of publications using statistics in 1985 (79.9%) was almost double the number used in 1975 (43.1%), mainly because of the increased use of inferential statistics versus descriptive statistics.11 In 2003, AJODO published 205 articles; 134 were original articles. Of these, 4 articles did not use statistics, and the other 130 reported 284 statistical tests. Of the statistical tests, 10 were descriptive, 265 were inferential, and 9 were miscellaneous. The inferential tests included 163 parametric, 72 nonparametric, and 30 post hoc analyses.4 

In the era of EBD, it is important for orthodontists to have a greater understanding of complex research designs and statistics to effectively practice in the new millennium. The purpose of this study was to ascertain the number, type, and overall usage of statistics in AJODO articles published in 2008. These data could then be compared to data for 1975, 1985, and particularly 2003. (A listing and description of statistical terms and tests appeared in a previous publication [p 515].4)

All articles published in the hard copy AJODO in 2008 (ie, did not include the “online only” articles) were categorized as original versus other types. The frequency and distribution of statistics used in the original articles for 2008 were then recorded in Microsoft Excel to be analyzed and tallied. The statistical aspects of the articles were initially dichotomized into those using statistics and those not using statistics. Statistical procedures were broadly divided into descriptive statistics (mean, standard deviation, range, percentage) and inferential statistics (t test, analysis of variance [ANOVA]). Because descriptive statistics are subsumed under inferential statistics, the two are not inherently mutually exclusive. In this regard, it should be pointed out that that there was a difference in how the statistical data were recorded for years 1975 and 198511 versus how data were recorded for AJODO articles in 20034 and 2008. In 1975 and 1985,11 it was easy to discern the overall statistical test used for each article. The complexity of research designs and statistical tests for 20034 and 2008 made it practically impossible to deduce an overall statistical test for each article, so the data for 2003 and 2008 reflect more or less the number of hits of each statistical test per article rather than one unique statistical test per article. The exception, as previously explained, is for those articles categorized as using descriptive statistics; only articles exclusively using descriptive statistics were listed in that category.

Inferential statistics were divided into parametric (eg, Student's t-test, ANOVA, correlation, regression) and nonparametric (eg, χ²) procedures. The frequency and percentage of post hoc analyses were also determined. A miscellaneous category was also established for such statistical treatments as reliability.

Data for AJODO articles in 1985, 2003, and 2008 were juxtaposed. Descriptive statistics were used to make comparisons.

In 1975, 1985,11 2003,4 and 2008, AJODO published 72, 87, 134, and 141 original articles, respectively. As previously reported,11 the percentage of articles using statistics in 1985 (77.9%) was nearly double that in 1975 (43.1%). There were a number of differences between the AJODO articles published in 2003 and 2008 and those published in1985 (Tables 1 through 3). In total, 205 articles were published in 2003 and 235 in 2008 (Table 3). There were 71 non-original articles published in 2003 and 94 in 2008 (Tables 1 and 2). There were 87 original articles published in 1985, 134 in 2003, and 141 in 2008 (Table 3). Regarding the non-original articles, the number of reviews increased from 4 in 2003 to 18 in 2008, and the number of Clinician's Corner articles increased from 7 to 16 over those same years. By 2008, a new section had been introduced in the journal: “Evidence Based Dentistry.” Two articles were written under that category in 2008, emphasizing an interest in evidence-based dental practice (Table 1).

Data from original AJODO articles published in 2003 and 2008 are juxtaposed in Table 2. From the 141 original articles published in the AJODO in 2008, 10 did not use statistics: five essays, three case reports, and two qualitative analyses (Table 2). In the remaining 131 articles, 286 statistical procedures were used (Table 2). Of these, 54 were descriptive statistical procedures, 186 were inferential statistical procedures, and 16 were grouped into the miscellaneous category. Of the descriptive tests, 19 were reported as general usage with no particular test recognized. Of the 186 inferential statistics, 141 were parametric procedures and 58 were nonparametric procedures. There were 16 post hoc analyses and 9 miscellaneous procedures, 2 of which were reliability tests (Table 2).

The most often used parametric, inferential tests were the Student's t-test (n  =  51), ANOVA (n  =  45), and correlation/regression (n  =  18) analyses. The most often used type of t-test was the general report (20 of 51). The most often cited ANOVA was the one-way ANOVA (28 of 45 procedures). The most often reported correlation/regression analysis was the Pearson analysis (10 of 18) (Table 2). The most often used nonparametric test was the χ² test (n  =  14), followed by the Mann-Whitney U test (n  =  13). The Tukey adjustment was the most reported post hoc analysis. The kappa analysis was the most often used reliability test.

A comparison of 2003 and 2008 revealed that some changes had taken place, some more significant than others. The number of original articles published increased from 134 to 141, and the number of articles using no statistics increased from 4 to 10 (Tables 2 and 3). The percentage of original articles using statistics in 1985 was 75.9%, compared with 94% in 2003 and 92.9% in 2008; original articles using statistics stayed relatively the same with a small 1.1% decrease from 2003 to 2008 (94% to 92.9%). The most unexpected change was an 8.5% increase in the number of articles solely using descriptive statistical analysis (ie, 7.1% in 2003 to 15.6% in 2008). From 2003 to 2008, in addition to the 8.5% increase in the number of articles using descriptive statistics, there was an 8.5% decrease in the number of articles using inferential statistics (from 92.9% to 84.4%; Table 2). The percentage of articles using inferential statistical analyses was 74.2% in 1985, 92.9% in 2003, and 84.4% in 2008 (Table 3).

The overall finding from this research project was that the number of AJODO articles using statistics generally increased from 75.9% to 94% and 92.9% in 1985, 2003, and 2008, respectively. The most significant change in AJODO publications in 2008 versus 2003 was an 8.5% increase in the use of descriptive statistics and an 8.5% decrease in the use of inferential statistics. There are several possible reasons for this finding. One explanation is that the increase in the publication of systematic reviews (and to a lesser degree this could apply to meta-analyses), which do not list statistical procedures (and would be placed in a no statistics category), lessened the number of articles placed in the inferential statistics category as defined for this research. Another explanation is the possibility that in or about 2003 the total number and complexity of statistical procedures were at a high point, and the difference in 2008 merely reflected a natural fluctuation and “regression toward the mean.” With this thinking in mind, one could predict that minor up and down swings will occur in upcoming years. An interesting question could be posed that if statistical analysis usage peaked in 2003, then shouldn't the focus be more on increasing the quality of the future research projects, that is, by including more RCTs? In addition, in the front of the journal Evidence-Based Dentistry12 has suggested that all journals include a method of qualitatively evaluating the level of evidence being reported.

There are at least two divergent views regarding the utility of knowledge of statistics for orthodontic practitioners. One notion is that orthodontists in the future will need to have little to no understanding of statistics. That is, with an exponential increase in the publication of systematic reviews, all any practitioner will need to do is electronically access the appropriate topic and/or clinical question and then click on an icon and the correct answer will automatically appear without much cognitive discernment. The contrasting view is that the orthodontist of the future will need more and not less knowledge of science, evidence, research design, and statistics. An important reason for this thinking is that there will never be enough systematic reviews in orthodontics to address the plethora of ever-changing clinical questions. New technologies and mechanisms to treat patients are always emerging. In addition, high-level studies and systematic reviews (Level 3 evidence) take many years to be assimilated. Based on the recent past, it appears that many more clinical decisions will need to be made based on Level 1 and 2 types of evidence. In this respect, a working knowledge and understanding of statistics will be of paramount importance considering the complexity of contemporary and future research designs.

The foundational principle for applying statistical tests is that sound basic research protocols have been followed. No statistical test can salvage poor-quality research. That is, it is possible that researchers have applied complicated and/or inappropriate statistics to overshadow poor research, so the reader must always be cautions of such occurrences even within peer-reviewed journal publications.

The findings from the present investigation could serve as a model for developing research and statistics curricula for graduate orthodontic programs and continuing education courses for practicing orthodontists. The information contained in this article could be developed into a rubric for learning statistics and reviewing published papers. A self-directed learning module could be developed based on the information gleaned from this investigation.

It is quite possible that a study similar to the present could be repeated every 5–10 years. Other orthodontic journals could be used in addition to the AJODO, such as the Angle Orthodontist, European Journal of Orthodontics, World Journal of Orthodontics, and so forth. Perhaps the statistical procedures from a comparable sample of articles used in the aforementioned orthodontic journals could be evaluated and the data compared and contrasted.

• In 1975, 1985, 2003, and 2008, AJODO published 72, 87, 134, and 141 original articles, respectively. The percentage of original articles using statistics was 43.1% in 1975, 75.9% in 1985, 94% in 2003, and 92.9% in 2008. Original articles using statistics stayed relatively the same with a small 1.1% decrease from 2003 to 2008 (94% to 92.9%)

• The percentage of articles using inferential statistical analyses was 74.2% in 1985, 92.9% in 2003, and 84.4% in 2008. From 2003 to 2008, there was an 8.5% increase in the number of articles using descriptive statistics (from 7.1% to 15.6%) and an 8.5% decrease in inferential statistics (from 92.9% to 84.4%).