[Taxacom] How good is peer review...

Sun Oct 6 03:53:43 CDT 2013

I think the situation is getting pretty bad as far as peer review is
concerned. I am aware of a number of manuscripts that were rejected from
low-impact taxonomic journals because they suffered from serious
fundamental errors, only to see them published - unchanged - a few months
later in "high-impact" journals. (I probably shouldn't name names, but what
the hell - Molecular Phylogenetics and Evolution, and PLOS ONE.) Obviously
the latter journals have not employed specialist reviewers ... quite likely
because they do not have the specialist taxonomy editors of purely
taxonomic journals. Regardless, taxonomy is fast entering a period where
impact is far more important than content, and taxonomic progress will
falter. My concern goes beyond Stephen's. Yes, sh!t will be published and
employers will be none the wiser. But who will be around to judge these
taxonomic "advances", to identify the shi!t? Fewer and fewer specialist
taxonomists are being employed. Emphasis is currently on high-impact output
over content, and technology over scholarship. Our science (and yes, I do
believe it is a science) is being dumbed down.

Tony

On Sun, Oct 6, 2013 at 7:22 PM, Stephen Thorpe
<stephen_thorpe at yahoo.co.nz>wrote:

> Yes, but the BIG problem is that you can make a career out of publishing
> sh!t, because employers don't wait for the long term to see how well one's
> publications hold up to future scrutiny. They, the fools, measure one's
> value by the number of (pre-pub.) peer reviewed publications that you can
> bang out ...
>
> Stephen
>
>
>
>
> ________________________________
>  From: Paul Kirk <P.Kirk at kew.org>
> To: Stephen Thorpe <stephen_thorpe at yahoo.co.nz>; "Weakley, Alan" <
> weakley at bio.unc.edu>; Bob Mesibov <mesibov at southcom.com.au>; TAXACOM <
> taxacom at mailman.nhm.ku.edu>
> Sent: Sunday, 6 October 2013 8:50 PM
> Subject: RE: [Taxacom] How good is peer review...
>
>
> In our domain - taxonomy - Post Publication Peer Acceptance is what
> matters. New names based on good taxonomy survive, names based on poor
> taxonomy sink into the oblivion of digital synonymy at the speed of light.
> Pre Publication Peer Review is becoming more of a myth than it ever was.
>
> Paul
> ________________________________________
>
> From: taxacom-bounces at mailman.nhm.ku.edu [
> taxacom-bounces at mailman.nhm.ku.edu] On Behalf Of Stephen Thorpe [
> stephen_thorpe at yahoo.co.nz]
> Sent: 06 October 2013 00:49
> To: Weakley, Alan; Bob Mesibov; TAXACOM
> Subject: Re: [Taxacom] How good is peer review...
>
> >Shouldn't we err on the side of promoting that ideas be published
> (=promulgated openly to the judgment of other scientists) by whatever
> means, and then judge them on their merit, over time?<
>
> Yes, though there needs to be a balance. The whole idea of peer review, as
> I understand it, was to set a MINIMUM standard on what gets published, i.e.
> if two RANDOMLY chosen peers think it is worth publishing, then it is
> probably worth publishing. Journals used to not want to publish crap, but
> it is less simple now that they measure themselves my impact factors and
> citation rates - some types of crap are good for these metrics! As with
> everything in life, there is no magic filter to filter out everything but
> the best work. Peer review was never designed to do this! Post publication
> peer review is a good idea, but I'm not sure that it will catch on ..
>
> Stephen
> ________________________________
> From: "Weakley, Alan" <weakley at bio.unc.edu>
> To: Bob Mesibov <mesibov at southcom.com.au>; TAXACOM <
> taxacom at mailman.nhm.ku.edu>
> Sent: Sunday, 6 October 2013 12:25 PM
> Subject: Re: [Taxacom] How good is peer review...
>
>
> So, here's _one_ possible distillation of this.
>
> Bad papers get published in all "systems".  Peer review vs. not, open
> access vs. paywall, print vs. e-journal -- all complex issues with no
> black-and-white.
>
> Perhaps a standard by which to judge all these controversies is what
> benefits or damages the advance of science?  I think exploitative journals
> publishing bad papers for a bribe (essentially) are morally reprehensible -
> but I'd be interested to know how much real damage or "impact" on the
> advancement of science they have.  I'll assert that we each in our own
> subfield can recognize real research from junk, and we ignore the junk.
> Certainly, if not ignored immediately, it falls by the wayside very fast.
> And, there is nothing new here - there has always been a lot of junk, and
> it has always been ignored.  Wheat and chaff, part of the business of
> science.  Modern digital world = lots more junk and lots more value:  sort
> it out!  So, in my opinion, they do little damage (though I would be glad
> to evaluate contrary evidence).  If, indeed, they do little real damage, we
> as a community need to be careful that any counter-measures (policing) do
> not
> cause more damage than the problem.
>
> Peer review is sometimes held up as a kind of holy preserver of quality.
> Yet, we all know that peer review is often done negligently and casually,
> and sometimes with actual scientific malfeasance:  the suppression of new
> radical ideas by the established experts who perceive a threat to their
> world view and their dominance.  The history of science is littered with
> examples.  Here's one {pardon a clip from Wikipedia!}:  "Lynn Margulis
> attended the University of Chicago, earned a master's degree from the
> University of Wisconsin-Madison in 1960, and received her Ph.D. in 1963
> from UC Berkeley. In 1966, as a young faculty member at Boston University,
> she wrote a theoretical paper entitled The Origin of Mitosing Eukaryotic
> Cells. The paper however was "rejected by about fifteen scientific
> journals," Margulis recalled".
>
> Even when not directly suppressive, peer review is fundamentally elitist:
> "well, that is an okay paper, but it doesn't warrant publication in OUR
> journal, which screens for only the most impactful papers".  But, are we
> really very good at judging that beforehand? {cue Margulis}".  Or, "that
> paper's by X, a leader in the field, so must be important and clearly
> publishable and important"; if the same paper came in from a 25-year old,
> would it receive the same review and decision?  So, there is a conservative
> and status-quo component to peer-review which can be antithetical to
> science at its best, admittedly balanced against its benefits.  A retort
> might be, "well, Margulis's fundamental new paradigm DID get out", but the
> appropriate re-retort is "how many major ideas didn't, or were (more)
> significantly delayed?"  Probably no easy answer to that.
>
> An additional point.  Perhaps we are all good at telling real scientific
> merit aside from any considerations of the author's ethnicity, gender,
> continent, institution, quality of English, or journal of publication - but
> I doubt it.  Note that this study used "fictitious authors ... affiliated
> with fictitious African institutions" created by "randomly combining
> Swahili words and African names with generic institutional words and
> African capital cities. My hope was that using developing world authors and
> institutions would arouse less suspicion if a curious editor were to find
> nothing about them on the Internet."  I think the real, tacit assumption
> here was that the African names and African institutions maximized the
> likelihood of skepticism about the quality of the paper and therefore if
> even despite that and its questionable content it was accepted, that was
> the strongest possible indictment of the journal's review policy.
>
> Perhaps an equally interesting study (and with appropriate control!) might
> have been to submit essentially similar (but more sophisticated and
> potentially scientifically valuable) papers to a reputable set of journals
> with different author ethnicities, genders, career stages, levels of fame
> or respect in the field, and institutions -- and judge how they were
> peer-reviewed (and accepted for publication or not) - I think there have
> been some studies showing gender bias.
>
> Shouldn't we err on the side of promoting that ideas be published
> (=promulgated openly to the judgment of other scientists) by whatever
> means, and then judge them on their merit, over time?
>
> Alan
>
> -----Original Message-----
> From: taxacom-bounces at mailman.nhm.ku.edu [mailto:
> taxacom-bounces at mailman.nhm.ku.edu] On Behalf Of Bob Mesibov
> Sent: Saturday, October 05, 2013 5:59 PM
> To: TAXACOM
> Subject: Re: [Taxacom] How good is peer review...
>
> Philipp Wagner wrote:
>
> 'In nomenclature the open access journals are not the great problem I
> think. A much more serious problem are online-only journals. With modern
> technology every private person is able to develop a journal and can write
> dubious ideas without any review. This is in fact the problem.'
>
> The problem is larger than private-person journals, as the spoof paper
> described in Science demonstrates. The ICZN moved on this problem and
> amended section 8 of the Code to allow electronic-only (valid) publishing
> only under certain conditions (see, for example,
> http://www.pensoft.net/journals/zookeys/article/3944/).
>
> Do you think these restrictions are inadequate?
> --
> Dr Robert Mesibov
> Honorary Research Associate
> Queen Victoria Museum and Art Gallery, and School of Agricultural Science,
> University of Tasmania Home contact:
> PO Box 101, Penguin, Tasmania, Australia 7316
> (03) 64371195; 61 3 64371195
> _______________________________________________
> Taxacom Mailing List
> Taxacom at mailman.nhm.ku.edu
> http://mailman.nhm.ku.edu/mailman/listinfo/taxacom
>
> The Taxacom Archive back to 1992 may be searched with either of these
> methods:
>
> (1) by visiting http://taxacom.markmail.org
>
> (2) a Google search specified as:  site:
> mailman.nhm.ku.edu/pipermail/taxacom  your search terms here
>
> Celebrating 26 years of Taxacom in 2013.
>

-- 
Dr Anthony C. Gill
Natural History Curator
A12 Macleay Museum
University of Sydney
NSW 2006
Australia.

Ph. +61 02 9036 6499
Editorial Board, Species and Systematics:
http://www.ucpress.edu/series.php?ser=spsy