Abstract

Objectives:

To assess the reporting quality of Cochrane and non-Cochrane systematic reviews (SR) in orthodontics and to compare the reporting quality (PRISMA score) with methodological quality (AMSTAR criteria).

Materials and Methods:

Systematic reviews (n  =  109) published between January 2000 and July 2011 in five leading orthodontic journals were identified and included. The quality of reporting of the included reviews was assessed by two authors in accordance with the PRISMA guidelines. Each article was assigned a cumulative grade based on fulfillment of the applicable criteria, and an overall percentage score was assigned. Descriptive statistics and simple and multiple linear regression analyses were undertaken.

Results:

The mean overall PRISMA score was 64.1% (95% confidence interval [CI], 62%–65%). The quality of reporting was considerably better in reviews published in the Cochrane Database of Systematic Reviews (P < .001) than in non-Cochrane reviews. Both multivariable and univariable analysis indicated that journal of publication and number of authors was significantly associated with the PRISMA score. The association between AMSTAR score and modified PRISMA score was also found to be highly statistically significant.

Conclusion:

Compliance of orthodontic SRs published in orthodontic journals with PRISMA guidelines was deficient in several areas. The quality of reporting assessed using PRISMA guidelines was significantly better in orthodontic SRs published in the Cochrane Database of Systematic Reviews.

INTRODUCTION

Systematic reviews, both Cochrane and non-Cochrane, are of increasing importance in guiding future medical research in addition to changing clinical practice and informing health policy. A comprehensive range of reviews has been conducted in dentistry, including a number of important reviews on fundamental orthodontic procedures and treatments.1 It is therefore critical that systematic reviews are conducted according to scientifically rigorous methodology2,3 and are clear and unbiased, which will facilitate their assessment and the translation of their findings into clinical practice. Reporting guidelines have been developed with the ultimate aim of improving the quality and content of systematic reviews, leading to more transparent results and estimates of treatment effects.

An 18-item checklist, the QUOROM Statement (Quality of Reporting of Meta-analyses), was developed in 1996 to improve the reporting of meta-analyses in health care research.4 This has since been superseded by the publication of the “Preferred Reporting Items for Systematic Reviews and Meta-analyses” (PRISMA) statement,5,6 comprising 27 specific items encompassing all aspects of the report. The key requirements of PRISMA include the accessibility of a protocol for the review, reporting of at least one complete electronic search, assessment of risk of bias in and across included studies, description of selective outcome reporting, reporting of limitations of the review and future research implications, and comment on sources of funding. Editors of orthodontic journals have been relatively efficient in implementing and endorsing reporting guidelines7,8 and most recently the PRISMA guidelines.8 

While reporting guidelines are integral to helping improve the quality of published manuscripts, it is often difficult to be certain of the scientific quality and rigor of the underlying research. Checklists have also been developed in an attempt to assess the quality of systematic reviews, as distinct from the reporting of the review process.9 Previous studies have been undertaken on orthodontic systematic reviews to investigate the quality of the research using AMSTAR criteria.10,11 It is unclear, however, whether reporting of systematic reviews is necessarily likely to reflect improved quality of the review process. The aims of this study were to assess the reporting quality of Cochrane and non-Cochrane systematic reviews published in leading orthodontic journals and the Cochrane Database of Systematic Reviews (CDSR) with reference to PRISMA guidelines. A secondary aim was to assess whether reporting quality (PRISMA score) is synonymous with the quality of the review process itself (AMSTAR score).

MATERIALS AND METHODS

A comprehensive literature search was undertaken to identify systematic reviews by searching five major orthodontic journals (American Journal of Orthodontics and Dentofacial Orthopedics [AJODO], Angle Orthodontist, European Journal of Orthodontics [EJO], Journal of Orthodontics, and Orthodontics and Craniofacial Research; February 2002 to July 2011) and the Cochrane Library from January 2000 to July 2011. The search process is outlined elsewhere.10 Reviews were considered eligible for inclusion if the terms systematic review or meta-analysis were in the title or abstract or if it was apparent in the text that a systematic review had been undertaken. Narrative reviews, surveys, historical reviews, and case reports with extensive literature reviews were excluded. Two authors screened potentially relevant articles independently, with disagreement resolved by discussion with a third author (N.P.); full-text articles were obtained for each potentially relevant study.

The quality of the reporting of the included reviews was assessed according to the level of compliance with the PRISMA guidelines. These guidelines incorporate 27 items pertaining to each section of the review, including title, abstract, introduction, methods, results, discussion, conclusion, and funding. Each criterion was assigned one of three grades: “no description,” “inadequate,” or “adequate.” Overall, each article was assigned a cumulative grade based on fulfilment of these 27 criteria. An overall percentage score was assigned to each review based on the sum of the applicable items; thus, if all 27 items were applicable, the total possible score was 81 (or 100%). Individual items were not applicable to several of the included reviews, and in these instances, these criteria were omitted and the denominator amended accordingly. Discrepancies between the authors in the grading of individual criteria were resolved by joint discussion. AMSTAR scores for the included studies have been reported in a prior study10; in this study, the relationship between overall reporting (PRISMA) and quality (AMSTAR) scores for each review was to be assessed.

Initially descriptive statistics on the characteristics of the included systematic reviews were undertaken. Modified PRISMA scores for each review were calculated and converted into a percentage. Modified PRISMA scores were approximately normally distributed; thus, parametric testing was undertaken including both simple and multiple linear regression analyses. Associations of reporting quality, with overall PRISMA score as the dependent variable, and independent predictors including journal of publication, time since publication, authorship country of origin, and number of authors were analyzed. The association between reporting of systematic reviews and the quality of the review process was also investigated using linear regression analysis. The level of statistical significance was prespecified at .05. Statistical analyses were performed with STATA version 12.0 software (Stata Corporation, College Station, Tex).

RESULTS

The distribution of the publications in each journal has previously been highlighted.10 Most reviews were published in AJODO (31.2%), in Angle Orthodontist (29.4%), and on the CDSR (23.9%). In terms of adherence to individual criteria, the overall performance was assessed as good for reporting of the abstract, explanation of rationale for the review, listing of information sources and criteria for study selection, analysis of risk of bias (or quality) of primary studies, presentation of results from individual studies, summary of the evidence, and reporting of both limitations and conclusions. However, other important steps in the review process, including registration of review protocol or reporting of funding sources, were undertaken only in Cochrane reviews. Similarly, reporting of risk of bias across studies, definition of summary measures, and explanation of methods analysis was considered adequate in only a minority of reviews (Table 1).

Table 1. 

Distribution of Scores for Individual Items on the 27-Item Modified PRISMA Checklist

Distribution of Scores for Individual Items on the 27-Item Modified PRISMA Checklist
Distribution of Scores for Individual Items on the 27-Item Modified PRISMA Checklist

The overall fulfilment of PRISMA criteria for articles published in individual journals, published in different geographical locations, comprising greater numbers of authors, and based on time since publication is provided in Table 2. The mean overall PRISMA score was 64.1% (95% confidence interval [CI], 62%–65%). Univariable analysis indicated that journal of publication and number of authors were significantly associated with the PRISMA score. Improved PRISMA scores were noted in reviews published in the CDSR (72.7%). In general, the reporting quality of reviews in each of the five orthodontic journals was not dissimilar (57% to 62.1%). Reporting appeared to be slightly better in AJODO (62.1%) and Angle Orthodontist (61.9%) than was the case for the three other journals. An increased number of authors were associated with an average 1.7% PRISMA score improvement with each additional coinvestigator; a mean reporting score in excess of 70% was reported when more than seven authors were involved. A weak trend toward improvement was noted with more recently published reviews, although the mean differences were minor.

Table 2. 

PRISMA Score Based on the Journal of Publication, Years Since Publication, Country of Authorship, and Number of Authorsa

PRISMA Score Based on the Journal of Publication, Years Since Publication, Country of Authorship, and Number of Authorsa
PRISMA Score Based on the Journal of Publication, Years Since Publication, Country of Authorship, and Number of Authorsa

Adherence to PRISMA guidelines related to journal of publication, continent of publication, number of authors, and time since publication were assessed using univariate and multivariate analyses, with reviews published in the CDSR and those published in Europe as the reference group for journal of publication and continent of publication, respectively (see Tables 3 and 4). Multivariable analysis demonstrated that the summary PRISMA score was related to journal of publication when both journal of publication and number of authors were considered simultaneously. The quality of reporting was considerably better in Cochrane systematic reviews (P < .001) than in non-Cochrane reviews published in all five of the specialty journals, with systematic reviews published in EJO having a 13.7% lower PRISMA score after accounting for the possible confounding effect of the number of authors. Similarly, both univariable (1.72; 95% CI, 0.96, 2.47; P < .001) and multivariable (0.79; 95% CI, 0.1, 1.48; P < .05) analysis suggested that adherence to PRISMA guidelines was likely to be improved as the number of authors increased. However, no statistically significant relationships were found between PRISMA scores and either continent of publication or length of time since publication. The association between AMSTAR score and modified PRISMA score was found to be highly statistically significant. A mean increase of 1 unit in AMSTAR score corresponds to an almost 3% (2.98%; 95% CI, 2.6, 3.36; P < .001) increase in modified PRISMA score.

Table 3. 

Mean PRISMA Score for Journal of Publication, Years Since Publication, Authorship Country, and Number of Authorsa

Mean PRISMA Score for Journal of Publication, Years Since Publication, Authorship Country, and Number of Authorsa
Mean PRISMA Score for Journal of Publication, Years Since Publication, Authorship Country, and Number of Authorsa
Table 4. 

Univariable and Multivariable Linear Regression–Derived Coefficients (β), 95% Confidence Intervals (CI), and P Values for Modified PRISMA Score as a Dependent Variable for Journal of Publication, Country of Authorship, Number of Authors, and Number of Years Since Publicationa

Univariable and Multivariable Linear Regression–Derived Coefficients (β), 95% Confidence Intervals (CI), and P Values for Modified PRISMA Score as a Dependent Variable for Journal of Publication, Country of Authorship, Number of Authors, and Number of Years Since Publicationa
Univariable and Multivariable Linear Regression–Derived Coefficients (β), 95% Confidence Intervals (CI), and P Values for Modified PRISMA Score as a Dependent Variable for Journal of Publication, Country of Authorship, Number of Authors, and Number of Years Since Publicationa

DISCUSSION

In a previous study, we assessed the quality of systematic reviews in orthodontics using AMSTAR criteria. This study differs in that it focuses primarily on reporting characteristics rather than review quality. Previous studies evaluating systematic reviews in medicine have examined the reporting compliance based on QUORUM criteria.12,13 The relatively recent introduction of PRISMA complicated the present review as it was developed in 2009 and endorsement in orthodontic journals typically lagged its initial publication by a further year. Nevertheless, it is hoped that the present study will provide a benchmark against which the reporting of further dental and orthodontic systematic reviews can be compared.

Previous research has estimated the ratio of non-Cochrane to Cochrane systematic reviews at 4∶1 based on a MEDLINE search.13 The relative proportion of either review type is similar in orthodontics, with the proportion of Cochrane reviews being slightly higher, although both Cochrane systematic reviews of orthodontic treatment and Cochrane systematic reviews considered relevant to orthodontics were included in the present study. Previous research has highlighted that the reporting quality of Cochrane reviews exceeded that of non-Cochrane reviews in both medical14,15 and dental16 research. This article is the first to substantiate these findings in orthodontic research. The mean percentage score for Cochrane reviews was more than 10% higher than for any of the specialty journals. This is not surprising as the Cochrane Collaboration sets stringent guidance on performance of systematic reviews that incorporates PRISMA guidelines. Furthermore, electronic publications such as Cochrane reviews are not constrained by limits on word counts, thereby permitting more complete reporting of the conduct of the review process; this may be a further reason for the enhanced reporting standard of these reviews.

Clearly, Cochrane reviews were generally reported to a very high standard and fulfilled most criteria in the majority of reviews that were included in this study. However, a systematic omission was noted in the failure to highlight the nature of the review in the title. It could, however, be argued that given the publication of these papers in the CDSR, the inclusion of the term systematic review in the title could be considered superfluous. It is therefore Cochrane policy to not include the words systematic review or meta-analysis in the review title. This was the only criterion in which Cochrane reviews were less compliant with PRISMA guidelines than non-Cochrane reviews.

Significant methodological weaknesses were identified in many of the non-Cochrane reviews, which are likely to have compromised interpretation and conclusions. In particular, there was no evidence of any prospective registration of the protocol in any of the non-Cochrane reviews. Registration of all systematic reviews has recently been advocated in electronic databases17 (eg, PROSPERO, Centre for Reviews and Dissemination, University of York, UK) to include a priori statements outlining review methodology that would more likely reduce the propensity for biased post hoc changes. While publication of protocols in up to 46% of reviews has been reported in the medical literature,13 this practice has obviously not filtered through into non-Cochrane orthodontic reviews. In addition, none of the non-Cochrane reviews disclosed any sources of funding, as opposed to Cochrane reviews in which an acknowledgment of any sources of funding and declarations of conflicts of interest are mandatory. Clear delineation of objectives and eligibility criteria using the PICOS acronym has been advocated in the PRISMA guidelines. Distinct and unambiguous criteria are instrumental in developing a relevant search strategy and in study selection. Ideally, these eligibility criteria should also refer to inclusion of studies with specific outcomes to permit assessment of the possibility of reporting bias due to selective reporting of results.18 Consequently, it is important that the reporting of eligibility criteria be improved in non-Cochrane orthodontic reviews, as 36% were found to be inadequate in the present study, with likely implications for the review process, selection of studies, results, and conclusions.

The search strategy was generally found to be satisfactory, although several of the reviews (17%) failed to provide full details of at least one electronic search. Similar shortcomings were identified in a previous assessment of orthodontic systematic reviews,2 with problems identified including restriction of searches to a single database (typically MEDLINE), omission of names and dates of database searches, involvement of only a single reviewer, and confinement of searches to English-language databases. The central problem related to a competent and comprehensive search is the possibility of omitting relevant articles, resulting in erroneous and possibly biased conclusions.2 Failure to report detailed searches also reduces transparency, preventing verification of the results of the review process. An important reason for failure to undertake comprehensive searches among dental review authors may relate to a perceived lack of return for additional effort.3 In view of the pivotal role played by systematic reviews in the provision of health care, journal editors and reviewers should not countenance inadequate search methods; improved reporting of search strategies in orthodontic systematic reviews should now be considered obligatory.

While little temporal change in the reporting quality was noticed in this study, it may be considered premature to expect the introduction and wider acceptance of PRISMA to have resulted in a significant difference in overall reporting quality in orthodontics given that these guidelines were officially adopted by AJODO in only 2010 and other orthodontic journals including EJO have yet to endorse them fully. The precursor of PRISMA was the QUOROM Statement published in 1999 providing guidance for meta-analysis of randomized controlled trials4; improved reporting of these studies has since been confirmed in medical research. A further problem stems from the suitability of PRISMA guidelines for analysis of reporting of all systematic review subtypes. In particular, PRISMA was initially intended as a template to guide reporting of reviews of the effects of interventions; however, it has also been used to guide diagnostic systematic reviews despite its limitations in this respect. It is likely that reporting guidelines for other types of systematic reviews including those of diagnostic studies and economic analyses will emerge, particularly as other review types are given increasing priority by the Cochrane Collaboration.19 

Many orthodontic journals now stipulate strict adherence to PRISMA reporting guidelines, with the result that the reporting of reviews is often quality assured with submissions accompanied by completed PRISMA checklists highlighting compliance with individual criteria. While conduct and reporting of clinical trials are quite distinct, the reporting and underlying methodological quality of systematic reviews are likely to be interrelated. This was confirmed in the present study, which identified a significant association between the PRISMA and AMSTAR scores. This positive association between AMSTAR scores and PRISMA reporting is reassuring and indicates that adherence to PRISMA guidelines is most likely to result in both the enhanced reporting and improved methodological quality of orthodontic reviews.

CONCLUSIONS

  • Reporting of orthodontic systematic reviews in leading orthodontic journals was found to be deficient in certain areas, particularly with respect to prospective registration of review protocols, reporting of sources of funding, assessment of reports of risk of bias across studies, definition of summary measures, and detailed explanation of the methods of analysis and eligibility criteria.

  • The quality of reporting assessed using PRISMA guidelines was significantly better in orthodontic systematic reviews published in the CDSR.

REFERENCES

REFERENCES
1.
Harrison
JE
,
O'Brien
KD
,
Worthington
HV
.
Orthodontic treatment for prominent upper front teeth in children
.
Cochrane Database Syst Rev
.
2007
;(
3
) :
CD003452
.
2.
Flores-Mir
C
,
Major
MP
,
Major
PW
.
Search and selection methodology of systematic reviews in orthodontics (2000–2004)
.
Am J Orthod Dentofacial Orthop
.
2006
;
130
:
214
217
.
3.
Major
MP
,
Warren
S
,
Flores-Mir
C
.
Survey of systematic review authors in dentistry: challenges in methodology and reporting
.
J Dent Educ
.
2009
;
73
:
471
482
.
4.
Moher
D
,
Cook
DJ
,
Eastwood
S
,
Olkin
I
,
Rennie
D
,
Stroup
DF
.
Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement
.
Lancet
.
1999
;
354
:
1896
1900
.
5.
Moher
D
,
Liberati
A
,
Tetzlaff
J
,
Altman
DG
,
for the
PG
.
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement
.
BMJ
.
2009
;
339
:
b2535
.
6.
Liberati
A
,
Altman
DG
,
Tetzlaff
J
,
et al .
The PRISMA statement for reporting systematic re- views and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration
.
BMJ
.
2009
;
339
:
b2700
.
7.
Turpin
DL
.
CONSORT/QUOROM guidelines adopted for reporting randomized clinical trials and systematic reviews [editorial]
.
Am J Orthod Dentofacial Orthop
.
2005
;
128
:
681
686
.
8.
Turpin
DL
.
Updated CONSORT and PRISMA documents now available
.
Am J Orthod Dentofacial Orthop
.
2010
;
137
:
721
722
.
9.
Shea
BJ
,
Grimshaw
JM
,
Wells
GA
,
et al . et al
Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews
.
BMC Med Res Methodol
.
2007
;
15
:
7
10
.
10.
Fleming
PS
,
Seehra
J
,
Polychronopoulou
A
,
Fedorowicz
Z
,
Pandis
N
.
Cochrane and non-Cochrane reviews in leading orthodontics journals: a quality paradigm
?
Eur J Orthod
.
2012
Apr
24
, [Epub ahead of print].
11.
Papageorgiou
SN
,
Papadopoulos
MA
,
Athanasiou
AE
.
Evaluation of methodology and quality characteristics of systematic reviews in orthodontics
.
Orthod Craniofac Res
.
2011
;
14
:
116
137
.
12.
Al Faleh
K
,
Al-Omran
M
.
Reporting and methodologic quality of Cochrane neonatal review group systematic reviews
.
BMC Pediatr
.
2009
;
9
:
38
.
13.
Moher
D
,
Tetzlaff
J
,
Tricco
AC
,
Sampson
M
,
Altman
DG
.
Epidemiology and reporting characteristics of systematic reviews
.
PLoS Med
.
2007
;
4
:
e78
.
14.
Lundh
A
,
Knijnenburg
SL
,
Jørgensen
AW
,
van Dalen
EC
,
Kremer
LC
.
Quality of systematic reviews in pediatric oncology: a systematic review
.
Cancer Treat Rev
.
2009
;
35
:
645
652
.
15.
Mrkobrada
M
,
Thiessen-Philbrook
H
,
Haynes
RB
,
Iansavichus
AV
,
Rehman
F
,
Garg
AX
.
Need for quality improvement in renal systematic reviews
.
Clin J Am Soc Nephrol
.
2008
;
3
:
1102
1114
.
16.
Ijaz
M
,
Croucher
RE
,
Marinho
VCC
.
Systematic reviews of topical fluorides for dental caries: a review of reporting practice
.
Caries Res
.
2010
;
44
:
579
592
.
17.
Clarke
M
,
Stewart
L
.
PROSPERO: the new international prospective register of systematic reviews. Cochrane Methods
.
Cochrane Database Syst Rev
.
2011
; ((
suppl 1
) ):
1
40
.
18.
Dwan
K
,
Altman
DG
,
Arnaiz
JA
,
et al . et al
Systematic review of the empirical evidence of study publication bias and outcome reporting bias
.
PLoS ONE
.
2008
;
3
((
8
) ):
e3081
.
19.
The Cochrane Collaboration
,
Cochrane reviews of diagnostic test accuracy
. .