Documentos de Académico
Documentos de Profesional
Documentos de Cultura
http://breast-cancer-research.com/supplements/12/S4/S13
S H O R T CO M M U N I C AT I O N
If peer review was a drug it would never be allowed onto takes place before a study is published. But what happens
the market, says Drummond Rennie, deputy editor of the after publication can also be called peer review, and that,
Journal Of the American Medical Association and I believe, is the peer review that really matters - the
intellectual father of the international congresses of peer process whereby the world decides the importance and
review that have been held every four years since 1989. place of a piece of research. Arthur Balfour, a British
Peer review would not get onto the market because we prime minister, might have been speaking of science
have no convincing evidence of its benets but a lot of when he famously said that nothing matters much and
evidence of its aws. few things matter at all. Many studies are never cited
Yet, to my continuing surprise, almost no scientists once, most disappear within a few years, and very few
know anything about the evidence on peer review. It is a have real, continuing importance.
process that is central to science - deciding which grant And the correlation between what is judged important
proposals will be funded, which papers will be published, in pre-publication peer review and what has lasting value
who will be promoted, and who will receive a Nobel seems to be small. Fabio Casati, professor of computer
prize. We might thus expect that scientists, people who science at the University of Trento, the holder of 20
are trained to believe nothing until presented with patents, and the founder of a liquid journal that had
evidence, would want to know all the evidence available dispensed with prepublication peer review, says:
on this important process. Yet not only do scientists Weve.found that peer review doesnt work, in the
know little about the evidence on peer review but most sense that there seems to be very little correlation
continue to believe in peer review, thinking it essential between the judgement of peer reviewers and the fate of
for the progress of science. Ironically, a faith based rather a paper after publication. Many papers get very high
than an evidence based process lies at the heart of marks from their peer reviewers but have little eect on
science. the eld. And on the other hand, many papers get average
ratings but have a big impact [1].
What is peer review? Indeed, the correlation could even be inverse in that
Peer review is not easily dened, and every grant giving peer review may well be biased against the truly original.
body and journal will have a process that is unique in I return to this point below.
some way. It is clearly something to do with an external, But what is peer review for? (And from now on I shall
third party reviewing a grant proposal or manuscript. But mean pre-publication peer review when I write just peer
how many external reviewers should there be? And under review. I will also be writing mostly about peer review of
what conditions should they review? Should they be manuscripts for publication rather than of grants because
anonymous or identied to authors and readers? And that is what has been studied the most, it is what I know
who is a peer? Somebody who also researches on the best, and it does have a clear alternative - simply
subject of the proposal or manuscript or somebody who publishing the manuscript and letting the world decide.) I
is simply in the same discipline? Should reviewers be see four main objectives for peer review: selecting what
trained? Dierent answers to these questions and many should be published, improving what is published,
others lead to wide variation in systems of peer review. detecting errors, and detecting fraud.
One useful way of classifying peer review of completed
studies is into pre-publication and post-publication. Is peer review effective?
When people speak and write about peer review they The Cochrane Collaboration, the organization that
usually mean pre-publication review, the process that through its systematic reviews produces the most reliable
evidence in medicine and health care, has reviewed the
*Correspondence: richardswsmith@yahoo.co.uk evidence on peer review of manuscripts and of grant
35 Orlando Road, London SW4 0LD, UK proposals. This is its conclusion on peer review of
manuscripts: At present, little empirical evidence is
2010 BioMed Central Ltd 2010 BioMed Central Ltd available to support the use of editorial peer review as a
Smith Breast Cancer Research 2010, 12(Suppl 4):S13 Page 2 of 4
http://breast-cancer-research.com/supplements/12/S4/S13
mechanism to ensure quality of biomedical research [2]. that all of the above phenomena are common. This is
And here is its conclusion on peer review of grant pro- surely a scandal [9].
posals: There is little empirical evidence on the eects of While Drummond Rennie writes in what might be the
grant giving peer review. No studies assessing the impact greatest sentence ever published in a medical journal:
of peer review on the quality of funded research are There seems to be no study too fragmented, no
presently available [3]. hypothesis too trivial, no literature citation too biased or
Of course the absence of evidence and evidence of too egotistical, no design too warped, no methodology
absence of eect are not the same thing, and many, too bungled, no presentation of results too inaccurate,
particularly the many with a vested interest in peer too obscure, and too contradictory, no analysis too self-
review, continue to believe that peer review is benecial serving, no argument too circular, no conclusions too
but that it has not been studied in the right way. Many triing or too unjustied, and no grammar and syntax
can also tell anecdotes of how a study they published was too oensive for a paper to end up in print.
much improved by peer review. Many can also, however,
tell anecdotes of bad experiences of peer review, and The downside of peer review
particularly of huge delays caused by peer review with no We have little or no evidence that peer review works, but
benet. Everybody could perhaps agree that it is shameful we have lots of evidence of its downside.
that a process so central to science should have no Firstly, it is very expensive in terms of money and
evidence to support its eectiveness - even if in reality it academic time. At the British Medical Journal we calcu-
is eective. lated that the direct cost of reviewing an article was, on
If peer review is to be thought of primarily as a quality average, something like 100 and the cost of an article
assurance method, then sadly we have lots of evidence of that was published was much higher. These costs did not
its failures. The pretentiously named medical literature is include the cost of the time of the reviewing academics,
shot through with poor studies. John Ioannidis has who were not paid by the journal. The Research Infor-
shown how much of what is published is false [4]. The mation Network has calculated that the global cost of
editors of ACP Journal Club search the 100 top medical peer review is 1.9 billion [10]. The cost in time is also
journals for original scientic articles that are both enormous, and many scientists argue that time spent
scientically sound and important for clinicians and nd peer reviewing would be better spent doing science.
that it is less than 1% of the studies in most journals [5]. The cost in time and money is much increased by
Many studies have shown that the standard of statistics in studies working their way down the food chain of journals.
medical journals is very poor [6]. A study may be submitted to Nature and rejected, then
Sadly we have many examples of studies published in sent to the New England Journal of Medicine and rejected,
medical journals that are not only scientically poor but and so on through the Lancet, British Medical Journal,
also have done great damage. The most famous example and several specialist journals before ending up in a local
is the Lancet paper that suggested that the MMR journal. Often the same reviewers will be consulted
(measles, mumps, rubella vaccine) caused autism: the repeatedly. And we know that if authors persist long
result was a drop o in the number of children vacci- enough, you can get anything published.
nated, epidemics of measles, and more than a decade of This expensive and time consuming process might be
fruitless argument [7]. Another example is the New acceptable if it sorted the information eectively, with
England Journal of Medicine article that seemed to show the most important studies being in the most important
that a new drug for arthritis, rofecoxib, was safer than the journals. Not only does this not happen (see below) but
traditional non-steroidal anti-inammatory drugs because this ineective sorting of information introduces an
it was less likely to cause gastrointestinal bleeding [8]. important bias - because the sexier articles end up in the
Unfortunately, the awed paper hid the increase in top journals. The many people who read these journals
myocardial infarctions. The paper was important in the because they think that they are reading what is most
new drug being widely used and in causing thousands of important are actually being presented with a distorted
patients to have heart attacks. view of science.
Doug Altman, perhaps the leading expert on statistics Secondly, peer review is slow. The process regularly
in medical journals, sums it up thus: What should we takes months and sometimes years. Publication may then
think about researchers who use the wrong techniques take many more months. A friend of mine, a fellow of the
(either wilfully or in ignorance), use the right techniques Royal Society, has written a paper that I think very
wrongly, misinterpret their results, report their results important for global health. As I write, it is still un-
selectively, cite the literature selectively, and draw published after two years of being reviewed by several
unjustied conclusions? We should be appalled. Yet top journals. None of the reviewers have raised a major
numerous studies of the medical literature have shown aw with the study.
Smith Breast Cancer Research 2010, 12(Suppl 4):S13 Page 3 of 4
http://breast-cancer-research.com/supplements/12/S4/S13
Thirdly, peer review is largely a lottery. Multiple studies prize, Solomon Bersons discovery of radioimmunoassay,
have shown how if several authors are asked to review a which led to a Nobel prize, and Bruce Glicks identi-
paper, their agreement on whether it should be published cation of B lymphocytes [15].
is little higher than would be expected by chance [11]. A Finally, peer review can be all too easily abused.
study in Brain evaluated reviews sent to two neuroscience Reviewers can steal ideas and present them as their own
journals and to two neuroscience meetings [12]. The or produce an unjustly harsh review to block or at least
journals each used two reviewers, but one of the meetings slow down the publication of the ideas of a competitor.
used 16 reviewers while the other used 14. With one of These have all happened. Drummond Rennie tells the
the journals the agreement among the journals was no story of a paper he sent, when deputy editor of the New
better than chance while with the other it was slightly England Journal of Medicine, for review to Vijay Soman
higher. For the meetings the variance in the decision to [16]. Having produced a critical review of the paper,
publish was 80 to 90% accounted for by the dierence in Soman copied some of the paragraphs and submitted it
opinions of the reviewers and only 10 to 20% by the to another journal, the American Journal of Medicine.
content of the abstract submitted. This journal, by coincidence, sent it for review to the boss
A fourth problem with peer reviews is that it does not of the author of the plagiarised paper. She realised that
detect errors. At the British Medical Journal we took a 600 she had been plagiarised and objected strongly. She
word study that we were about to publish and inserted threatened to denounce Soman but was advised against
eight errors [13]. We then sent the paper to about 300 it. Eventually, however, Soman was discovered to have
reviewers. The median number of errors spotted was two, invented data and patients and left the country.
and 20% of the reviewers did not spot any. We did further
studies of deliberately inserting errors, some very major, Improving peer review
and came up with similar results. Peer review is often compared with democracy in being
The fth problem with pre-publication peer review is the least bad system available, and attempts have been
bias. There have been many studies of bias - with made to improve peer review - by blinding reviewers to
conicting results - but the most famous was published the identity of authors, opening up the process so that
in Behavioural and Brain Sciences [14]. The authors took authors and possibly even readers know the identity of
12 studies that came from prestigious institutions that the reviewers, and training reviewers. In summary, none
had already been published in psychology journals. They of these methods have made much dierence [17,18].
retyped the papers, made minor changes to the titles,
abstracts, and introductions but changed the authors Alternatives to pre-publication peer review
names and institutions. They invented institutions with For journal peer review the alternative is to publish
names like the Tri-Valley Center for Human Potential. everything and then let the world decide what is impor-
The papers were then resubmitted to the journals that tant. This is possible because of the internet, and Charles
had rst published them. In only three cases did the Leadbeater has illustrated how we have moved from a
journals realise that they had already published the paper, world of lter then publish to one of publish then lter
and eight of the remaining nine were rejected - not and a world of I think to one of We think [19]. The
because of lack of originality but because of poor quality. problem with ltering before publishing, peer review, is
The authors concluded that this was evidence of bias that it is an ineective, slow, expensive, biased, inecient,
against authors from less prestigious institutions. Most anti-innovatory, and easily abused lottery: the important
authors from less prestigious institutions, particularly is just as likely to be ltered out as the unimportant. The
those in the developing world, believe that peer review is sooner we can let the real peer review of post-publica-
biased against them. tion peer review get to work the better.
Perhaps one of the most important problems with peer Fabio Casati puts it thus: If you and I include this paper
review is bias against the truly original. Peer review might in our journals [our personal collections], we are giving it
be described as a process where the establishment value.When this is done by hundreds of people like us,
decides what is important. Unsurprisingly, the establish- were using the selection power of the entire community
ment is poor at recognizing new ideas that overturn the to value the contribution. Interesting papers will rise
old ideas. It is the same in the arts where Beethovens late above the noise. This is we think rather than what a few
string quartets were declared to be nothing but noise and arbitrarily selected reviewers think.
Van Gogh managed to sell only one painting in his The problem of nding an alternative to peer review of
lifetime. David Horrobin, a strong critic of peer review, grants is more dicult - because clearly there are not the
has collected examples of peer review turning down resources to fund every grant proposal. But it may be
hugely important work, including Hans Krebss descrip- more important to try and nd an alternative - such as
tion of the citric acid cycle, which won him the Nobel giving highly successful scientists funds to pursue what
Smith Breast Cancer Research 2010, 12(Suppl 4):S13 Page 4 of 4
http://breast-cancer-research.com/supplements/12/S4/S13
they want - because the anti-innovatory nature of peer of the supplement are available online at http://breast-cancer-research.com/
review may mean that important science does not get supplements/12/S4