A key principle in the federal Information Quality Act and its accompanying government-wide guidelines is the disclosure of sufficient information that qualified third parties can reproduce the results using the same data and methods. Though this is only a minimum requirement for influential information, it is frequently not met. Federal agencies routinely disseminate influential information without providing the building blocks third parties need to reproduce results.
The August 28, 2015 issue of Science includes a paper shows that published scientific papers in respected psychology journals fail this test. Only 39% of the time could the consortium reproduce statistically significant effects. On average, these effects were only half as large.
The article summarizes research conducted by a 270-member consortium of scientists. (Less technical summaries of the work can be found in the National Post, Popular Science, Washington Post, New York Times, and Science Daily.)
In 2005, Stanford’s John Ioannides published the most widely cited study like this, titled “Why most published research findings are wrong.” (Google Scholar reports that it has been cited more than 3,000 times in scholarly papers.)
In regulatory settings, agencies often rely on a small number of scientific studies to develop estimates of risk and regulatory benefit. Rarely are these studies reproduced; even when a subsequent study is published that appears to corroborate the original, it may be because both studies relied on the same data and therefore are not actually independent.