Science relies on peer review, but peer review is a tool of limited utility for discerning truthfulness. The editors of Science may have discovered this anew with the requested retraction of a path-breaking 2014 paper purporting to show that opposition to gay marriage is fairly easily reversed by gay canvassers.
A recent scientific study garnered enormous attention when it was published in December 2014. This occurred for two reasons. First, it confirmed the political judgments and policy views of mainstream reporters and editors. Second, and more importantly, it was published in Science, the flagship magazine of the American Academy of Arts and Sciences. Apparently the editors of Science also found the paper appealing for political and policy reasons; they published an “editor’s summary” along with the paper.
Recently, this study was alleged to have been constructed from fake data. On May 20, 2015, Science issued an “editorial expression of concern,” noting that second (and senior) author Donald Green (Columbia University) has asked that the paper be retracted. Primary author Michael LaCour stands by his findings and has committed to provide a response to Science by May 29. [UPDATE 5/28/15: Science today issued a retraction.]
Our interest is not to debate the substance of the article or the political and policy issues surrounding it. Rather, this episode is a useful reminder of the limits of scholarly peer review. Those limits are always present but only rarely recognized. The reasons why are fairly simple.
Peer review has two dominant functions within the business of scholarship. First, it is the method by which scholarly journals ration scarce pages. Every reputable journal has a page limit, and the editor’s task is to select from the available manuscripts those papers that maximize value to the journal.
Second, the publication of scholarship in peer-reviewed journals is the principal metric colleges and universities use to evaluate faculty quality, for example for tenure decisions and salary offers. The number of articles published, but publications are weighted by journal quality. Very few scholarly journals have quality weights as high as Science, so it was inarguably a huge professional coup for primary author Michael LaCour, a doctoral candidate at UCLA. (If the paper is retracted, however, that coup could have a disastrous impact on his pending appointment at Princeton University.)
Scholarly peer review also serves a separate, distinct, and wholly undesirable function. It enables everyone — reporters, editors, Members of Congress and the President, regulators, the public, and even scientists themselves — to commit the logical fallacy of appealing to authority. It is much easier to infer that a paper is correct because it was published in a scholarly journal than to carefully review the paper yourself. That is especially so with high-quality journals like Science. Journal editors succumb to this fallacy other ways, such as by improvidently appealing to the authority of respected second authors who, perhaps surprisingly, “took care not to analyze any primary data.” When a study confirms what one wants to be true, the attraction of this logical fallacy may become overwhelming.
Appeals to a journal’s authority are misplaced for many reasons, but at the top of the list should be the fact that journal peer reviewers only rarely bother to validate whether a manuscript’s empirical results are correct. Rarer still is the peer reviewer who investigates whether an author’s data are valid and reliable, or worse, fabricated. Whatever the perils may be of appealing to authority, is makes no sense if the authority to whom one appeals made no effort to validate the facts about which the appeal is made.
A peer reviewer’s assignment is to examine a manuscript’s technical quality, opine on its suitability for publication, suggest areas for improvement, and do these things in a few hours of uncompensated time. Regrettably, editors often ask peer reviewers to opine on whether the manuscript should be published. This is a question reviewers lack sufficient information to address because they do not know what other manuscripts are competing for the journal’s scarce pages.
Federal information quality guidelines and peer review guidelines give a rebuttable presumption of “basic” quality to scientific information published in peer-reviewed journals. That standard is actually easy to rebut, for example by a simple showing that information quality was not part of the charge to peer reviewers. More importantly, the “basic” quality standard is far below what is required of government-disseminated information with policy implications.
For Science, this incident offers yet another reminder that prestige is a wasting asset if not aggressively protected. Whenever the editors consider publishing manuscripts that reasonably can be expected to be cited in public policy debates, the editors should require authors to adhere to federal information quality standards as a requirement for publication.
LaCour, Michael J., and Donald P. Green. 2014. When Contact Changes Minds: An Experiment on Transmission of Support for Gay Equality. Science. 346(6215): 1366-1369.
Can a single conversation change minds on divisive social issues, such as same-sex marriage? A randomized placebo-controlled trial assessed whether gay (n = 22) or straight (n = 19) messengers were effective at encouraging voters (n = 972) to support same-sex marriage and whether attitude change persisted and spread to others in voters’ social networks. The results, measured by an unrelated panel survey, show that both gay and straight canvassers produced large effects initially, but only gay canvassers’ effects persisted in 3-week, 6-week, and 9-month follow-ups. We also find strong evidence of within-household transmission of opinion change, but only in the wake of conversations with gay canvassers. Contact with gay canvassers further caused substantial change in the ratings of gay men and lesbians more generally. These large, persistent, and contagious effects were confirmed by a follow-up experiment. Contact with minorities coupled with discussion of issues pertinent to them is capable of producing a cascade of opinion change.
Dialogue opens the door to attitude change
Personal contact between in-group and out-group individuals of equivalent status can reduce perceived differences and thus improve intergroup relations. LaCour and Green demonstrate that simply a 20-minute conversation with a gay canvasser produced a large and sustained shift in attitudes toward same-sex marriage for Los Angeles County residents. Surveys showed persistent change up to 9 months after the initial conversation. Indeed, the magnitude of the shift for the person who answered the door was as large as the difference between attitudes in Georgia and Massachusetts.