Interesting stories have appeared explaining how the gay marriage study published by Science was exposed as fraud. Previously we’ve noted that there was a failure in peer review, or more accurately, a failure to recognize the limits of what scholarly peer review can achieve.
To that we can now add a few insights into the limits of scholarship, at least as it is commonly practiced today.
Jesse Singal has a long article in New York Magazine’s Science of Us column summarizing an interview with David Broockman, the chief sleuth who uncovered the fraud. Below are a few of the lessons learned:
- Appeals to authority were endemic, not isolated, and they protected the fraud from being discovered.
Science practices double-blind peer review, meaning that the reviewers are not supposed to know the identity of the authors and vice versa. That may be true in theory, “but between earlier versions of papers floating around the internet and the fact that everyone knows what everyone else is working on, the reality is quite different.” First, much of today’s science involves large data sets collected, maintained and analyzed by large, well-funded research teams. If a peer reviewer does not know who is involved with these projects, that reviewer may not know the subject matter well enough to be an effective peer reviewer.
Second, it is actually quite easy for peer reviewers to learn the identity of a manuscript’s author(s) even if they try not to do so. The list of references often gives it away, self-citation being an incurable and highly transmissible disease among academics.
If the author(s) are respected in their field, peer reviewers may reduce the intensity of effort they commit to the task. Reasons are varied. Sometimes it is simply yet another appeal to authority. But sometimes it reflects a reticence, or even fear, of taking on respected scholars who because of others’ appeals to their authority can fairly easily publish problematic research.
- Correcting others’ errors may be a vital function of science, but it’s not one that scientific institutions reward.
To be successful, a young researcher on tenure-track needs to stake out unclaimed territory in his field and publish with mad abandon. Sometimes that will involve overturning some well established principle or widely shared belief among the field’s practitioners. Sometimes it involves proving that another researcher was simply wrong. It’s best if that other researcher is already dead.
This is why the practice of correcting scientific error is usually taken up by scholars whose careers do not require tenure, or who already have it and bountiful free time to pursue such things. It means that academia is the last place to look for error correction.
- Challenging others’ work is professionally dangerous for untenured professors
As a graduate student, Broockman clearly recognized this risk and treated it with great respect. He was repeatedly advised not to go public with his concerns because doing so could damage his career. One mentor is quoted as having told him, “Even if you have uncontroversial proof, you still shouldn’t do it.”
[T]he moment your name is associated with the questioning of someone else’s work, you could be in trouble. If the target is someone above you … you’re seen as envious, as shamelessly trying to take down a big name. If the target is someone at your level, you’re throwing elbows in an unseemly manner. In either case, you may end up having one of your papers reviewed by the target of your inquiries (or one of their friends) at some point…
Science may be worshipped as an objective pursuit for truth, but that pursuit is practiced by people, and people have less admirable motives.
- Fraud could have been detected many places along the way, but at each such location there were institutions, norms and systems in place that prevented truth from escaping.
Among the numerous waypoints, two stand out. The first was when Donald Green failed to review the data and methods before agreeing to be a co-author. Singal reports that Green admits he failed to exercise appropriate supervisory judgment as the senior author. (In his letter to Science asking that the paper be retracted, he tried to absolve himself of that responsibility by hiding behind institutional rules.) Had Green insisted on reviewing the data and methods, and the perpetrator of the fraud shared them, it is plausible but not certain that the fraud would have been detected.
The second key waypoint was Science. Its editors have not disclosed what went wrong in its peer review process. (It has instead denied that anything went wrong, which raises serious questions about how many other papers it has published are erroneous or fraudulent.) From what has been disclosed so far, it appears that the editors committed at least two distinct errors. First, they succumbed to the fallacy of appealing to authority, in this case, the reputation of the co-author, who says he never reviewed the primary author’s data and methods. Second, the editors lowered their evidentiary standards because they wanted the results of this study to be true. As Singal writes:
LaCour’s impossible-seeming results were treated as truth, in part because of the weight Green’s name carried, and in part, frankly, because people — researchers, journalists, activists — wanted to believe them (emphasis in original).
Ideology has no legitimate place in scientific publication, but in fact it is quite common. The editors of many scholarly journals have fairly explicit ideological, political or policy views; sometimes, the very subject matter of the journal is ideological, political or policy-driven. It’s no surprise, then, when they publish tendentious research. But Science is supposed to be above that, and this incident shows that it isn’t.
IDEAS FOR REFORM
- Post-publication peer review.
Broockman suggests there is a role for this, and that is basically what he and his colleagues did. Indeed, never before in the history of science has it been as easy to conduct post-publication peer review as it is today. The problem is scholarly institutions — universities, journals and grantmakers alike — discourage it. These institutions would need to change their incentive structures to reward post-publication review, an that seems highly unlikely.
- Publication of data and computer code.
Post-publication peer review would be much easier if journals insisted that manuscripts accepted for publication include online access to their data and computer code. Not only would this make the task of validation easier, it would discourage authors from submitting fraudulent manuscripts.
Many authors refuse or resist disclosure because they consider their data proprietary — even if their research was publicly funded. They can certainly choose not to disclose, and submit to journals that do not require disclosure, but journals desiring a reputation for high quality should set higher standards for transparency and reproducibility.
This would be consistent with federal information quality standards, which require science disseminated by the government to be transparent, reproducible and objective. That requires access to the data and computer code. Effective implementation of these standards — that is, effectively forbidding the dissemination of noncompliant science — would deter a great deal of scientific error, fraud, and general mischief.
- Rewarding scholars who uncover error.
Universities and journals can play an important role by incentivizing error detection, but it conflicts so much with the cultures of these institutions that their participation seems unlikely.
Grantmakers (perhaps especially, federal grantmakers) could devote a nontrivial fraction of total funding for the detection and correction of error (particularly perhaps error in papers published based on their prior funding). It’s one thing for universities and journals to express their willingness to reward error correction, but quite another for there to be sufficient funding to conduct the necessary research. Until there is funding, error correction will remain the province of eclectic scholars (like Neutral Source), risk-taking graduate students, and those highly motivated to expose the incompetence or venality of their intellectual rivals.
A more promising route is the establishment of journals devoted to uncovering scientific error. Blogs like Retraction Watch, which provided the forum for exposing fraud in this case, are indispensable precursors. But they are inherently reactive, summarizing in one place journals’ acknowledgement of error. They are not substitutes for rigorously peer-reviewed correction of error. As long as scholars are judged on the number of their peer-reviewed publications, there is no substitute for a peer-reviewed Journal of Scholarly Error Correction.