The August 2003 issue of Mental Retardation (MR) contained a peer-reviewed article by James Conroy, Scott Spreat, Anita Yuskaukas, and Martin Elks entitled, “The Hissom Closure Outcomes Study: A Report on Six Years of Movement to Supported Living.” Shortly after publication of Conroy et al. (2003), Kevin Walsh and Theodore Kastner contacted the editor of MR expressing concerns about the article in view of alleged discrepancies between that article and an unpublished 1996 report by Conroy.
Walsh and Kastner subsequently asked Conroy to provide the data on which Conroy et al. was based. Despite months of communications among Walsh and Kastner, Conroy, Oklahoma state officials, and the MR Editor, the matter had not been resolved. Although the American Association on Mental Retardation (AAMR) Board had previously enacted a policy on data-sharing regarding articles published in its journals, the MR Editor advised Walsh and Kastner that the policy did not include enforcement mechanisms, and he did not have the authority to require Conroy et al. to furnish their data.
Walsh and Kastner then brought their concerns regarding access to Conroy et al.'s data to the AAMR Board. In September 2004, the Board enacted a revised policy on data-sharing. The policy is important and warrants reproduction here:
It is the policy of AAMR that any author publishing a professional paper in an AAMR journal agrees to follow APA guidelines with respect to sharing of raw data.
Specifically, the author(s) agree to retain raw data for at least 5 years following publication of a paper, and to provide others access to raw data if they receive a formal (written or email) request to provide the raw data needed to verify a published finding.
Note: Within this policy, authors are not obligated to provide raw data for use other than to verify analyses reported in a professional paper published in an AAMR journal, and at no point is there an expectation that sharing of raw data would occur in a form that violated informed consent, privacy rights, or ethical standards associated with protection of human subjects.
If an author does not respond to a formal request for raw data within a three-month period, it is appropriate for the person making the request to contact the editor of the AAMR journal that published the initial paper and for the editor to contact the author(s) and clarify AAMR policy.
If after three months from contact by the editor, the author(s) continues to withhold raw data requested for verification, then the journal editor may notify all authors of the article in question that for a period of three years, no manuscript will be considered for acceptance in that AAMR journal that includes them as a co-author. The journal editor will notify the AAMR Board President, AAMR Executive Director, other AAMR journal editor(s) and AAMR Publications Committee Chair prior to notifying the authors. (Enacted September 2004)
In Fall 2004, the MR Editor wrote Conroy to request compliance with AAMR's policy. In January 2005, Conroy complied by sending a data disk. As Fujiura said in his commentary, the peer-review process is ultimately based on trust that investigators have adequately and faithfully represented their data and findings. Therefore, it is the responsibility of authors of peer-reviewed empirical articles to furnish their data for independent review and analysis to any other researcher upon request. Conroy complied with the requirements of AAMR's policy, but he could have been more forthcoming and responsive.
A possible defense of Conroy's reluctance to share data with Walsh and Kastner is that they are ideological opponents. Walsh and Kastner, on the one hand, and Conroy, on the other, have appeared as expert witnesses on opposite sides in court cases on deinstitutionalization. It also is very clear that Kastner and Walsh have scrutinized Conroy's research very carefully (see, for example, Kastner, 2000; Taylor, 2000). The motivations of Walsh and Kastner in requesting data from Conroy are irrelevant. An ethical and professional responsibility is an ethical and professional responsibility.
In March 2005, a manuscript, “Fact and Fiction: A Critical Review of Conroy et al. (2003),” was submitted to the journal by Walsh and Kastner. The editor invited the associate editor in charge of the Conroy et al. submission, Marty Wyngaarden Krauss, to participate in the review process. We agreed that the reviewers of this submission should be persons who have expertise in statistical analysis and who could provide fair reviews. The reviewers were provided with the Walsh and Kastner critique, the original Conroy et al. article, the 1996 report on Hissom, and the data disk furnished by Conroy.
The reviews we received were outstanding, among the best we have ever seen in our editorial experience. The reviewers not only reviewed the critique and the original Conroy et al. article, but also the Conroy et al. data disk. The reviewers' comments on Conroy et al. were troubling. The kindest things that could be said about the article was that their selection of the subsample for their analysis was not well-defined, that their statistics were simplistic, and that they did not account for a Type I error. Yet, each of the reviewers expressed concerns about the tone of the critique and its gratuitous attack on the personal integrity of Conroy et al.
We invited Walsh and Kastner to submit a revised version of their manuscript. This critique is published in this issue of the MR. We also sent Conroy edited versions of the reviewers' comments and offered him the opportunity to publish an addendum to Conroy et al. addressing the reviewers' concerns and questions. We regret that he did not respond to this invitation.
We asked the three original reviewers of the Walsh and Kastner critique to submit commentaries on Walsh and Kastner and Conroy et al. David Felce and Glenn Fujiura have waived their anonymity as reviewers of the original Walsh and Kastner submission and have provided the extremely thoughtful and well-reasoned commentaries that follow.
Readers of the journal should pay careful attention to Felce and Fujiura in this controversy. Fujiura (2006) wrote,
Community living is not an independent variable. Where people live and are supported is a policy choice. It is a value. My bias is that all should be served in the community, but I also know the community must do much better. (p. 374)
Felce (2006) concluded,
Developing a greater understanding of which factors within which types of provision arrangements promote which quality of life outcomes is essential if people with intellectual disabilities are to have the opportunity to lead more fulfilling and desirable lives than they do currently. (p. 381)
The peer-review process is not infallible. Published articles may contain errors that should have been caught during the review process. Reviewers typically do not have access to the raw data on which studies are based and must rely on the competence and honesty of investigators to report their results accurately.
Walsh and Kastner have raised serious concerns about the credibility of Conroy et al. When concerns are raised about the credibility of an article published in a journal, it is the responsibility of the editors to take these concerns seriously; encourage the open sharing of data; invite rigorous review; and, most of all, strive for openness and transparency in the editorial decision-making process. By doing these things, editors can overcome the imperfections of the peer-review process.
The flaws in Conroy et al. identified by Walsh and Kastner and confirmed by independent reviewers are serious. It is our opinion that Conroy et al. (2003) should not be cited as informing policy discussions and debates in the professional and research literature.