I cannot give any scientist of any age better advice than this: the intensity of the conviction that a hypothesis is true has no bearing on whether it is true or not. The importance of the strength of our conviction is only to provide a proportionately strong incentive to find out if the hypothesis will stand up to critical evaluation.
Sir Peter Medawar (1979), Winner of the Nobel Prize in Medicine, 1960
Having served as associate editors of this Journal for more than a decade, we have been pleased to participate in its growth and development. The Journal now is generally regarded as the premier peer-reviewed forum for discussions of policy and practice in the field of intellectual and developmental disabilities. The editor and his staff, associate editors, consulting editors, and contributors have created an important reference for practitioners and policymakers around the world. An important step in the Journal's maturation came in 2000, when the peer review processes and editorial procedures were augmented by the addition of a policy on data-sharing, requiring authors submitting articles to make their data available, upon request, to researchers for independent analysis (Kastner, 2000; Taylor, 2000). This policy was recently strengthened by the Board of Directors of the American Association on Intellectual and Developmental Disabilities (AAIDD) by the inclusion of sanctions for authors who do not share data under the terms outlined in the policy (Taylor & Krauss, 2006). These two steps are needed to ensure that articles printed in the Journal can be independently replicated, an important element of sound science, especially when used as a basis for policy. With the availability of these tools, independent researchers, together with the editorial staff, can now verify the accuracy of claims made by authors in manuscripts published in our Journal. However, one unresolved question remains: What should be the response when, after independent review, a published article is subsequently found to not meet the standards required for publication?
In this perspective we recommend that the Board of Directors of AAIDD implement a policy giving editors for both AAIDD Journals (the other being the American Journal on Mental Retardation) the authority to remove published articles that are subsequently found to not be supported by the data used to prepare the original manuscript or that may be, in some way, fraudulent. We propose that editors be given the power to remove articles when they have been found to contain substantial errors and pose the risk of inaccurately reflecting “knowledge” secondary to the supposed validity conferred by publication in our Journals. In our view, the removal of articles would rarely be necessary. Removal should be reserved for only the most serious of circumstances, primarily when research fraud is involved or when articles contain problems that are likely to substantially mislead readers as to their basic conclusions. Similarly, removal of articles may be necessary to prevent fraudulent or faulty work from being cited by later authors (cf. Sox & Rennie, 2006).
This Journal is currently faced with such an article. In the October 2006 issue of Mental Retardation, we published a critique of Conroy, Spreat, Yuskauskas, and Elks (2003) that was accompanied by two commentaries from independent peer reviewers as well as a comment by the editor and an associate editor (Felce, 2006; Fujiura, 2006; Taylor & Krauss, 2006; Walsh & Kastner, 2006). In the published commentaries, the two reviewers agreed with our conclusions that there were substantial problems in representation of the data used to reach the conclusions contained in Conroy et al. (2003). Scientific errors in the paper were serious enough to compromise many, if not most, of the findings and interpretations by the authors. In their accompanying editorial, the editors noted that the problems in the article were “confirmed by independent review and are serious,” resulting in the following statement: “It is our opinion that Conroy et al. (2003) should not be cited as informing policy discussion and debates in the professional research literature” (Taylor & Krauss, 2006, p. 371).
Although we agree with this recommendation, the failure to remove the article from the scientific literature could be taken to mean that the concerns lie only with the conclusions reached by the original authors. However, it is clear from the evidence that the article failed to meet certain scientific standards and that the authors' conclusions were not supported by the data presented. As such, we believe that a discussion of the viability of this article should be focused on whether it should be removed from the Journal entirely rather than be left as a published article, albeit one that may have generated some disagreement with the authors' conclusions. Although the policy opinions offered in Conroy et al. (2003) may be valid, readers simply cannot make a judgment about many of them based upon the research that was performed.
A reasonable person might then ask: “Shouldn't an article with ‘serious’ problems be formally removed from the Journal so that it could not be cited in the future by other authors unfamiliar with its deficiencies?” In the case of AAIDD Journals, the answer to that question appears to be that the Board of Directors has never formally established policies on the removal or retraction of articles after publication. Despite the fact that the editors (Taylor & Krauss, 2006) recommended that the Conroy et al. (2003) article not be cited as “informing policy discussion and debates,” there is nothing to prevent such citations by authors of articles in the future. Additionally, preventing such citations may be difficult because the critique and the accompanying editorial were published 3 years after the original article. Thus, without a proper removal or retraction notice that can be fully indexed by search services, computerized searches of the literature are likely to turn up the article and authors are likely to cite it, regardless of the recommendation in the 2006 editorial. Furthermore, such a caveat does little to prevent the article from being cited in various other contexts that were contained in the article (i.e., nonpolicy elements), such as the outcomes of a number of the specific measures reported. At present, such citations would be possible even though it was demonstrated that many of these measures were erroneous as reported. We are, therefore, concerned that later citations in the literature could easily serve to “validate” anew the problematic findings for other readers in other contexts. In our view this would be an unfortunate outcome that would mislead readers of this Journal, and perhaps others as well.
A historical example seems relevant here. Garfield and Welljams-Dorof (1990) studied the citations of work carried out by Stephen E. Breuning and his colleagues. Breuning, then at the University of Pittsburg, published a number of articles between 1980 and 1984 in which they examined the effects and side effects of certain drugs on aspects of hyperactivity in children with intellectual and cognitive disabilities. As noted by Garfield (1990), the “Breuning case is especially interesting to clinicians because it involved research on patients and influenced at least one state's policies on care for the mentally retarded” (p. 88). In the mid-1980s, a mentor of Breuning exposed this work as containing faked data. Ultimately, a National Institute of Mental Health report declared that none of the studies had been carried out as reported and, finally, in 1988, Breuning pleaded guilty to making false statements on federal grant applications. Although researchers eventually avoided citing the Breuning work, it took a number of years and wide publicizing of this case before citations declined. In fact, as Garfield and Welljams-Dorof showed, others' citations of Breuning articles actually increased during the 1980s, peaking at about the time the fraud was exposed, but continuing, albeit at a lower rate, for some years thereafter. Although these authors concluded that, over time, the scientific literature appears to purge itself of clearly fraudulent work, one wonders if this will be the case for erroneous data that are published in articles in which no fraud is alleged and no specific retraction notice is created (i.e., Conroy et al., 2003). In fact, we worry that it is quite possible that the editorial caveat will have little, if any, effect on the citation of the Conroy et al. (2004) article with respect to policy issues.
Although there is no evidence of fraud in the current case, it seems clear that data management and analysis errors resulted in false data making their way into Conroy et al. (2003). Unfortunately, the authors did not take advantage of the editor's offer to have them explain the data and clarify the results of their study (Taylor & Krauss, 2006). Regardless of the genesis of problems in the research, it is our view that our present article should be addressed by the editor and the Board. There are several sources of guidance for the Board of AAIDD in responding to Conroy et al. (2003).
The Department of Health and Human Services, Public Health Service (PHS) Policies on Research Misconduct (2005) identified three primary types of research misconduct: (a) fabrication, (b) falsification, and (c) plagiarism. Although the PHS definition excludes honest errors, the definition of falsification of data includes “data or results such that the research is not accurately represented in the research record” (Department of Health, and Human Services, 2005, p. 28386). Clearly, the data errors and incorrect findings of Conroy et al. (2003), as shown in the critique and commentaries appearing in the October 2006 issue of this Journal, are consistent with this definition.
Furthermore, important professional associations require authors to voluntarily retract deficient articles from publication or to vouch for their data. For example, the 1934-9556-45-6-408-APA1Ethical Guidelines of the American Psychological Association—APA (2007) contains the statement that: “If psychologists discover significant errors in their published data, they take reasonable steps to correct such errors in a correction, retraction, erratum, or other appropriate publication means” (APA Ethical Guidelines, 8.10.b).
Similarly, policies of the Journal of the American Medical Association require at least one author of an article to take “responsibility for the integrity of the data and the accuracy of the data analyses” with this exact statement required in the Acknowledgment section of manuscripts (American Medical Association, 2007; also cf. Davidoff et al., 2001). No doubt similar policies or statements could be identified in other bodies of research ethics and publication policies.
Although voluntary retraction would solve the problem in this case, the authors of Conroy et al. (2003) have not sought to retract their article nor have they responded to a request by this Journal's editor to respond in print to the published critique (Taylor & Krauss, 2006). Therefore, unfortunately, the article remains a bona fide published article of this Journal, despite the fact that it contains known errors in nearly all of its aspects and, based on excellent peer review, has been found to be flawed to such a degree that the editor and an associate editor have recommended that it not be cited in the future. It is for these reasons that we assert that to protect the integrity of AAIDD journals, the Board of Directors should enact a policy to allow the removal of such articles when necessary and that, in this instance, the editor remove Conroy et al. (2003) from the journal.
This request reflects the increasingly complex research and publishing environment. In a noteworthy editorial in the New England Journal of Medicine, the editors of 13 leading medical journals expressed their fears that the scientific objectivity of published research was threatened in an intellectual environment in which various other factors, such as economic and ideological concerns, compete with scientific objectivity (Davidoff et al., 2001). In relation to the current issue, we are concerned that because the general thrust of Conroy et al. (2003) is in keeping with what we have referred to as the zeitgeist surrounding the community imperative (Walsh & Kastner, 2006), the critical review of this article will be overlooked. It is here that we find value in the words of Peter Medawar, with which we opened this Perspective, and we urge readers to draw from this quote the lesson that, as researchers, support for our convictions needs to be derived from empirical findings rather than empirical findings flowing from our convictions.
Finally, we are concerned that certain observers have interpreted our efforts to maintain the quality of this Journal as somehow being ideologically biased against community integration. This is simply not the case. Indeed, we support deinstitutionalization and community integration or any other effort aimed at improving the quality of lives of those served; furthermore, we support affording consumers, along with their families and guardians, the information they need to make informed choices from an array of meaningful service options. However, what we do not support, in fact, what we are opposed to, is poorly executed science. The two of us have worked together, exclusively in community settings, for nearly 2 decades to improve the quality of services and supports available to persons with intellectual and developmental disabilities. Our efforts have been instrumental in preventing hundreds of individuals from being treated in inappropriate care settings, including emergency rooms, hospitals, nursing homes, and other institutional settings. We currently direct a program providing health care, care management, and habilitative supports to nearly 4,000 persons with intellectual and developmental disabilities living in communities throughout New Jersey (Kastner & Walsh, 2006). Furthermore, our work with local, state, and public or private stakeholders has typically increased the quality of care and the range of options available to support persons with intellectual and developmental disabilities in settings of their choice.
In fact, our professional efforts have made significant contributions to the smooth and successful closure, or contraction, of developmental centers in New Jersey. For example, when the North Princeton Developmental Center closed in New Jersey in 1998, our organization, Developmental Disabilities Health Alliance, developed a health care center in Mercer County to provide ongoing services, including primary care, seizure management, and mental health services to former residents of the facility. We have no doubt that individuals moving to the community from the North Princeton Developmental Center would have encountered significant barriers to obtaining adequate health care were it not for the services we have provided in the state. In the end, we understand that the community versus institution debate adds little to our understanding of the nature of the quality of life that most people experience, and we support the pursuit of new research agendas (e.g., Felce, 2006) that are focused on the quality of life of all individuals with intellectual and developmental disabilities regardless of their living situation.
Nonetheless, to be published and to become part of the knowledge base of science, research findings need to be able to meet the requirements of scientific rigor to the greatest extent possible and, when found to be substantially in error, need to be clarified or removed from this knowledge base. We believe that the proposal we have made herein will help the Journals of AAIDD in the development of such a knowledge base. Therefore, we applaud the editors of the AAIDD Journals for the work they do; we commend the reviewers of our critique of Conroy et al. (2003) for the comprehensiveness of their reviews; we implore the Board of Directors of AAIDD to enact a policy that allows its Journal editors to remove articles from the Journals when necessary; and, under such a policy, we ask the editor of this Journal to remove Conroy et al. (2003) from Intellectual and Developmental Disabilities (formerly Mental Retardation).
Kevin K. Walsh, PhD (email@example.com), Director of Quality Management and Research, Developmental Disabilities Health Alliance, 1527 Forest Grove Rd., Vineland, NJ 08360-1865. Theodore A. Kastner, MD, Associate Professor of Clinical Pediatrics, New Jersey Medical School, University of Medicine and Dentistry of New Jersey, and President, Developmental Disabilities Health Alliance, 1285 Broad St., Bloomfield, NJ 07003-3045