[Taxacom] How good is peer review...

[Paul Kirk]

In our domain - taxonomy - Post Publication Peer Acceptance is what matters. New names based on good taxonomy survive, names based on poor taxonomy sink into the oblivion of digital synonymy at the speed of light. Pre Publication Peer Review is becoming more of a myth than it ever was.

Paul
________________________________________
From: taxacom-bounces at mailman.nhm.ku.edu [taxacom-bounces at mailman.nhm.ku.edu] On Behalf Of Stephen Thorpe [stephen_thorpe at yahoo.co.nz]
Sent: 06 October 2013 00:49
To: Weakley, Alan; Bob Mesibov; TAXACOM
Subject: Re: [Taxacom] How good is peer review...

>Shouldn't we err on the side of promoting that ideas be published (=promulgated openly to the judgment of other scientists) by whatever means, and then judge them on their merit, over time?<

Yes, though there needs to be a balance. The whole idea of peer review, as I understand it, was to set a MINIMUM standard on what gets published, i.e. if two RANDOMLY chosen peers think it is worth publishing, then it is probably worth publishing. Journals used to not want to publish crap, but it is less simple now that they measure themselves my impact factors and citation rates - some types of crap are good for these metrics! As with everything in life, there is no magic filter to filter out everything but the best work. Peer review was never designed to do this! Post publication peer review is a good idea, but I'm not sure that it will catch on ..

Stephen
________________________________
From: "Weakley, Alan" <weakley at bio.unc.edu>
To: Bob Mesibov <mesibov at southcom.com.au>; TAXACOM <taxacom at mailman.nhm.ku.edu>
Sent: Sunday, 6 October 2013 12:25 PM
Subject: Re: [Taxacom] How good is peer review...


So, here's _one_ possible distillation of this.

Bad papers get published in all "systems".  Peer review vs. not, open access vs. paywall, print vs. e-journal -- all complex issues with no black-and-white.

Perhaps a standard by which to judge all these controversies is what benefits or damages the advance of science?  I think exploitative journals publishing bad papers for a bribe (essentially) are morally reprehensible - but I'd be interested to know how much real damage or "impact" on the advancement of science they have.  I'll assert that we each in our own subfield can recognize real research from junk, and we ignore the junk.  Certainly, if not ignored immediately, it falls by the wayside very fast.  And, there is nothing new here - there has always been a lot of junk, and it has always been ignored.  Wheat and chaff, part of the business of science.  Modern digital world = lots more junk and lots more value:  sort it out!  So, in my opinion, they do little damage (though I would be glad to evaluate contrary evidence).  If, indeed, they do little real damage, we as a community need to be careful that any counter-measures (policing) do not
 cause more damage than the problem.

Peer review is sometimes held up as a kind of holy preserver of quality.  Yet, we all know that peer review is often done negligently and casually, and sometimes with actual scientific malfeasance:  the suppression of new radical ideas by the established experts who perceive a threat to their world view and their dominance.  The history of science is littered with examples.  Here's one {pardon a clip from Wikipedia!}:  "Lynn Margulis attended the University of Chicago, earned a master's degree from the University of Wisconsin-Madison in 1960, and received her Ph.D. in 1963 from UC Berkeley. In 1966, as a young faculty member at Boston University, she wrote a theoretical paper entitled The Origin of Mitosing Eukaryotic Cells. The paper however was "rejected by about fifteen scientific journals," Margulis recalled".

Even when not directly suppressive, peer review is fundamentally elitist:  "well, that is an okay paper, but it doesn't warrant publication in OUR journal, which screens for only the most impactful papers".  But, are we really very good at judging that beforehand? {cue Margulis}".  Or, "that paper's by X, a leader in the field, so must be important and clearly publishable and important"; if the same paper came in from a 25-year old, would it receive the same review and decision?  So, there is a conservative and status-quo component to peer-review which can be antithetical to science at its best, admittedly balanced against its benefits.  A retort might be, "well, Margulis's fundamental new paradigm DID get out", but the appropriate re-retort is "how many major ideas didn't, or were (more) significantly delayed?"  Probably no easy answer to that.

An additional point.  Perhaps we are all good at telling real scientific merit aside from any considerations of the author's ethnicity, gender, continent, institution, quality of English, or journal of publication - but I doubt it.  Note that this study used "fictitious authors ... affiliated with fictitious African institutions" created by "randomly combining Swahili words and African names with generic institutional words and African capital cities. My hope was that using developing world authors and institutions would arouse less suspicion if a curious editor were to find nothing about them on the Internet."  I think the real, tacit assumption here was that the African names and African institutions maximized the likelihood of skepticism about the quality of the paper and therefore if even despite that and its questionable content it was accepted, that was the strongest possible indictment of the journal's review policy.

Perhaps an equally interesting study (and with appropriate control!) might have been to submit essentially similar (but more sophisticated and potentially scientifically valuable) papers to a reputable set of journals with different author ethnicities, genders, career stages, levels of fame or respect in the field, and institutions -- and judge how they were peer-reviewed (and accepted for publication or not) - I think there have been some studies showing gender bias.

Shouldn't we err on the side of promoting that ideas be published (=promulgated openly to the judgment of other scientists) by whatever means, and then judge them on their merit, over time?

Alan

-----Original Message-----
From: taxacom-bounces at mailman.nhm.ku.edu [mailto:taxacom-bounces at mailman.nhm.ku.edu] On Behalf Of Bob Mesibov
Sent: Saturday, October 05, 2013 5:59 PM
To: TAXACOM
Subject: Re: [Taxacom] How good is peer review...

Philipp Wagner wrote:

'In nomenclature the open access journals are not the great problem I think. A much more serious problem are online-only journals. With modern technology every private person is able to develop a journal and can write dubious ideas without any review. This is in fact the problem.'

The problem is larger than private-person journals, as the spoof paper described in Science demonstrates. The ICZN moved on this problem and amended section 8 of the Code to allow electronic-only (valid) publishing only under certain conditions (see, for example, http://www.pensoft.net/journals/zookeys/article/3944/).

Do you think these restrictions are inadequate?
--
Dr Robert Mesibov
Honorary Research Associate
Queen Victoria Museum and Art Gallery, and School of Agricultural Science, University of Tasmania Home contact:
PO Box 101, Penguin, Tasmania, Australia 7316
(03) 64371195; 61 3 64371195