When Science Gets It Wrong

When science gets it wrong

Let the light shine in

FROM THE ECONOMIST
Two big recent scientific results are looking shaky—and it is open peer review on the internet that has been doing the shaking

SCIENTISTS make much of the fact that their work is scrutinised anonymously by some of their peers before it is published. This “peer review” is supposed to spot mistakes and thus keep the whole process honest. The peers in question, though, are necessarily few in number, are busy with their own work, are expected to act unpaid—and are often the rivals of those whose work they are scrutinising. And so, by a mixture of deliberation and technological pressure, the system is starting to change. The internet means anyone can appoint himself a peer and criticise work that has entered the public domain. And two recent incidents have shown how valuable this can be.
The first concerns pluripotent stem cells, the predecessors of every other body cell. Pluripotent cells interest doctors and biologists, who hope to use them to investigate diseases, test drugs and, eventually, regrow patients’ damaged body parts.
At first, such cells had to be extracted from embryos, an ethically dodgy process if those embryos were human. Then a way of making them from skin cells was invented. This, though, involves fiddling about with proteins called transcription factors, and is finicky. So when, in January,Nature published a pair of papers describing a simpler way of making them, people around the world sat up. In these papers Haruko Obokata of the RIKEN Centre for Developmental Biology in Kobe, Japan, and her team, claimed to be able to create pluripotent cells by exposing ordinary, non-stem cells to weak acids, physical squeezing and some bacterial toxins.
No sooner had the papers been published than doubts emerged. Others tried to replicate Dr Obokata’s methods and failed—and made their findings known on the web. Other blogs and websites published apparent irregularities in diagrams and pictures in the papers, and pointed out that one passage seemed to have been copied verbatim from elsewhere.
In April a RIKEN committee concluded that Dr Obokata had twice manipulated her data in an intentionally misleading fashion, something they classed as research misconduct. The more light that was shone on the papers, the weaker the claims became. Although Dr Obokata had defended her work, her co-authors were divided on whether formally to retract the papers. On June 4th she reportedly agreed to withdraw both (although as The Economistwent to press both were still available, unchanged, on Nature’s website).
Shrinking BICEPs
The second claim came from cosmology. On March 17th researchers from the Harvard-Smithsonian Centre for Astrophysics, led by John Kovac, held a press conference at which they announced that they had discovered interesting patterns in the cosmic microwave background, a type of weak radiation left over from the universe’s earliest moments. They said they had spotted the signatures of primordial gravitational waves, ripples in space formed just after the Big Bang.
Once again, it was big news (including in The Economist). The existence of such waves would give strong support for the theory of inflation, which holds that the early universe underwent a brief burst of faster-than-light expansion. Inflation was put forward in the 1980s by theorists as a way to resolve various knotty problems with the standard theory of the Big Bang. But although it is widely assumed to be true, direct evidence that it happened had been lacking.
Dr Kovac and his colleagues made much of their data available online at the time, prompting hundreds of physicists to check their work. Doubts soon surfaced. The team’s claim to have spotted the waves relies on them having diligently scrubbed out every possible source of false positives. But doing that is hard, because the most likely culprit—interstellar dust—is poorly understood. Such diligence is made doubly difficult by the fact that, although several teams are hunting for primordial gravity waves, the glory of being the first to spot them means none is willing to share its data with the others.
The various online arguments culminated with the publication of an online paper by researchers from New York University, Princeton University and the Institute for Advanced Study (also in Princeton). This concluded that Dr Kovac’s data, which came from an Antarctic telescope called BICEP-2, may well have been contaminated by space dust, and that the purported gravitational waves may be much weaker than the team first claimed—if they exist at all.
No one is accusing Dr Kovac and his colleagues of the sort of sharp practice for which Dr Obokata was censured. Instead, the argument is about whether their results were really solid enough to justify the rapturous reception they were given, says Katherine Mack, an astrophysicist at the University of Melbourne. But both cases reflect the rise of open, post-publication review on Facebook and Twitter, by e-mail, on blogs, and in the comments sections of websites like arXiv, which hosts preprints of papers in physics and mathematics. As Paul Knoepfler, a biologist at the University of California, Davis, whose blog was used to co-ordinate the efforts of those trying to replicate Dr Obokata’s work, puts it, “I suspect that if published even five years ago, the [stem cell] papers’ serious problems would have gone unnoticed for far longer.”
The BICEP-2 results are an extreme example of this trend. They were announced without having first been published in a journal. This means that all the reviewing took place after the fact. Doing things this way makes some scientists nervous, especially when the subsequent discussion throws up problems. Blogs, comment threads and Facebook discussion groups mean that some of the debates which used to happen behind closed doors now take place in public, where anyone can watch.
The public, however, pay for most of this stuff. That open peer review gives them a glimpse into the reality of life inside the ivory tower is probably a good thing. Despite the activities of people like Dr Obokata, science is one of the most trustworthy human activities. But as Ronald Reagan put it in a different context, “Trust, but verify.”