[Taxacom] Open review as a wiki

[Stephen Gaimari]

A couple of issues I'd like to address. I'll start with a random thought, then a couple of specific points. 
 
1) With the 1000's of scientific journals out there, publishing all kinds of scientific research on a variety of topics, wouldn't it potentially hurt taxonomy to distance itself from mainstream science? A typical reader of one of these scientific journals would potentially NEVER read a taxonomic paper, and would NEVER gain an appreciation for the science of taxonomy. It seems to me that sequestering taxonomy would only serve to further isolate taxonomists, further make taxonomy appear as non-science, and further promote the idea that we are providing a simple service to real science. The perception will be that taxonomy can't be published in "highly ranked" journals. Yes, open access is a terrific thing (and should be for all kinds of science). Yes, good review is critical to good papers (that is true in all fields, but I don't see any other disciplines sequestering themselves). But it seems to me that the only people who would (only potentially) benefit from such isolation from the scientific community would be the taxonomists themselves. It certainly would not put taxonomy at the forefront of science where it belongs, but would further relegate it deeper into obscurity. It certainly would not promote the importance of taxonomy and the need for more taxonomists to be educated and hired in faculty positions. We should be actively promoting the idea that taxonomy is the foundation for ALL of biological science, not some pseudoscience tucked away in their own little world.
 
 
2) "bear in mind that  traditional reviews such as these only see a manuscript when the author(s) believe it is absolutely ready to publish!"
 
I've reviewed enough papers to know that even when the author(s) believe something is ready to publish that it may not be so. In my opinion, a reviewer should devote his precious time to a manuscript, aimed at assessing and improving it, when it is ready. Too much time is wasted on manuscripts that are clearly not ready for this stage in the publication process.
 
 
3) "Imagine an author who takes 4 years to get a manuscript ready for submission - if it takes another 4 months for it to get reviewed, revised, and printed, then that's great. Now, imagine that same author puts the first draft online for review right at the start .... etc."
 
I'm sorry - this made no sense to me. First, how many times do you expect an expert to review and rereview a manuscript? Does each specialist want to take the time to continually massage someone else's research? When does this constant improvement give authorship to the reviewer? And for this draft submission to act as a de facto specimen request? I don't think so. If someone is tackling a revision, they better know the community of experts in a group from whom they should request specimens, and which collections house important materials.

4) "many of the *nastiest* problems facing the taxonomic community can be traced to the lack of a universal standard of review"
 
Having such an open-review system would not create a universal standard of review in any way that I can see. Rather, it would promote more of a free-for-all, completely unstandardized. The review process, no matter whether it starts from the rough draft stage or after a manuscript is finished, will still be A) dependent upon the community interested enough in that group to work with the manuscript, B) dependent upon the individual personalities and opinions of those reviewers, and C) dependent upon the subjective opinion of the assigned "impartial referee" as to whether criticisms were adequately addressed.  
 
 
5) "Each incoming manuscript would have an impartial referee assigned to it, basically making sure the discussion stayed civil, that as many of the registered interested parties had made comments as seemed practical, and there would be a checklist of criteria that determined when something was ready to be accepted; most of these would be objectively-defined content (type deposition, etymology, non-homonymy, diagnoses, images/illustrations, etc.) that are BUILT INTO the template (so anything left blank would be glaringly obvious), but the crucial subjective decision would be "Are there any criticisms which have not been adequately addressed or refuted?". Ultimately, in such a system, I believe that the only manuscripts that would "languish" are those for which the author(s) failed to fill out the template completely in a timely manner, or could not (or would not) deal with criticisms adequately - remember, again, that the manuscripts would be online for a long, long time. If an author tries to conceal ongoing projects and refuses to submit them for review until they are "ready", then *those* authors may find the resulting review phase to be onerous, but only because they did not utilize the system in the manner it was intended."
 
Sorry - taxonomy is not a template science. Yes, there are certain elements that are critical such as those listed above, but an acceptable manuscript is not one that simply fits into the template. What a boring science this would be if it was a matter of filling in a template! Maybe we could all just fill in the blanks on a web-form, and that will be the paper submitted for review? (sorry - yes - that was sarcasm...). 
 
As for addressing reviewer criticisms - isn't this impartial referee the same thing as a journal editor, who weighs the criticisms and the authors responses when deciding on publication? It seems that this impartial referee will make the final decision of publication or rejection, based upon their subjective opinion of whether criticisms were adequately addressed. Back to this universal standard - the only standard thing is the template, which is governed anyway by the Codes of Nomenclature (i.e., requirements to become available).
 
Just a couple of cents worth.
Cheers,
Steve 
 
--------------------------------------------------------------------------------------
Dr. Stephen D. Gaimari
Program Supervisor (Entomology) &
Co-Curator, California State Collection of Arthropods

Plant Pest Diagnostics Lab
California Department of Food and Agriculture
3294 Meadowview Road
Sacramento, CA 95832-1448, USA

916-262-1131 (tel.)
916-262-1190 (fax)
sgaimari at cdfa.ca.gov 
http://www.cdfa.ca.gov/phpps/ppd/staff/sgaimari.html  
--------------------------------------------------------------------------------------

>>> Doug Yanega <dyanega at ucr.edu> 3/31/2008 6:02 PM >>>
Neal Evenhuis asked:

>At 10:29 AM -0700 3/31/08, Doug Yanega wrote:
>>the model some of us are advocating
>>would be OPEN review - real-time, online, non-anonymous - in the
>>fashion of a Wiki. When all the criticisms of a submitted work have
>>been dealt with (by accommodating the valid criticisms and dismissing
>>the inappropriate ones)
>
>This is an admirable dream. I've been dreaming this too but being an 
>editor for over 30 years I've come up with many questions relating 
>to problems that ultimately need the human factor. If you've got 
>some answers, let me/us know.
>
>1. How would it be accomplished to take in the vagaries of 
>reviewers' habits including not responding in a timely manner?

How do various journals presently accommodate? For most, they solicit 
reviews, ask for a commitment to a deadline, and therefore skip 
potential reviewers who can't commit. Most print journals probably 
have only a slightly longer turnaround time than Zootaxa, where 
(unless I'm mistaken) the major time-saving step is the vastly 
shorter interval between approval and publication. However, bear in 
mind that  traditional reviews such as these only see a manuscript 
when the author(s) believe it is absolutely ready to publish!

Imagine an author who takes 4 years to get a manuscript ready for 
submission - if it takes another 4 months for it to get reviewed, 
revised, and printed, then that's great. Now, imagine that same 
author puts the first draft online for review right at the start; 
with feedback occurring throughout the process, and the streamlining 
influence of having a template to work from, it would very likely NOT 
take four years to finish it - maybe it would take only two years, 
let's say - in which case a statement like "Oh my god, we can't 
possibly tolerate a review process that takes two years!" is 
comparing apples to oranges (in this case, the manuscript got into 
print FASTER as a result of online review). As I said earlier, the 
submission of a manuscript would be part of the process of making 
specimen requests - it is certainly not uncommon for a study to take 
years to complete, if the amount of material being examined is 
substantial (for an entomologist, it would not be unusual to have to 
look over thousands of specimens from dozens of institutions). If 
reviewers can make comments during the time the author is examining 
specimens, then that leaves a VERY long time window, and even the 
laziest reviewer can't say that a year or more is not enough time to 
find an opportunity to make comments.

Remember also that the open review process will involve potentially 
every conceivable qualified reviewer, in which case an additional 
question should be asked: "Which is better, (A) a paper which goes 
from initial submission (as a finished work) to print in two months 
but is seen by only three reviewers, and is printed with errors and 
problems which a larger group of reviewers could have detected and 
fixed BEFORE it was printed, or (B) a paper which goes from initial 
submission (as a draft) to print in a year but which has been 
scrutinized by dozens of reviewers, including all of the pertinent 
authorities?" Just ask yourself how many times you've seen something 
incorrect or inappropriate come out in print that you KNOW you could 
have prevented had you been involved in the review. Under this 
system, everyone would *have* that opportunity, and everyone would 
benefit!

I would argue that many of the *nastiest* problems facing the 
taxonomic community can be traced to the lack of a universal standard 
of review; I believe firmly that making the review process as lengthy 
and exhaustive as possible is a desirable goal, with benefits that 
far outweigh any perception of encumbrance. High-quality work that 
meets the highest standards will, by its nature, pass through the 
system quickly and easily; work of lower quality will be held up only 
just as long as is necessary to bring it up to the higher standards 
desired.

>2. What happens when there may only be one specialist who could give 
>a good review and he/she is out of email contact for four weeks 
>while in the field?

Four weeks is nothing when a manuscript is taking shape over a year 
or two. Surely, some time during the development of a work there will 
be an opportunity for everyone with an interest to make their 
opinions known to the author(s), other reviewers, and referee.

>3. How are time limits to review set (three weeks, a month?)-- or 
>does the manuscript languish online for months or years on end in 
>hopes "someone" will notice it? The web is passive after all.
>4. Will there be some sort of registration of interested parties so 
>that they can get email automatic notices when their speciality 
>taxon group comes up for review?
>5. How do you know that everyone that is a specialist in a 
>particular group knows a manuscript is open for review?

These three are linked: there *would* be registration of interested 
parties, with automatic notification. There would also be a 
backtracking feature that would indicate who has not seen their 
notifications yet. Each incoming manuscript would have an impartial 
referee assigned to it, basically making sure the discussion stayed 
civil, that as many of the registered interested parties had made 
comments as seemed practical, and there would be a checklist of 
criteria that determined when something was ready to be accepted; 
most of these would be objectively-defined content (type deposition, 
etymology, non-homonymy, diagnoses, images/illustrations, etc.) that 
are BUILT INTO the template (so anything left blank would be 
glaringly obvious), but the crucial subjective decision would be "Are 
there any criticisms which have not been adequately addressed or 
refuted?". Ultimately, in such a system, I believe that the only 
manuscripts that would "languish" are those for which the author(s) 
failed to fill out the template completely in a timely manner, or 
could not (or would not) deal with criticisms adequately - remember, 
again, that the manuscripts would be online for a long, long time. If 
an author tries to conceal ongoing projects and refuses to submit 
them for review until they are "ready", then *those* authors may find 
the resulting review phase to be onerous, but only because they did 
not utilize the system in the manner it was intended.

>6. Bookmarking the "Open Review" site as one's web browser home page 
>might solve some of these problems but in reality, someone who sets 
>their web browser home page to default to news of a bunch of 
>taxonomy manuscripts available for review online would be a VERY 
>weird person who doesn't have much of a life. Any solution to this?

Many times the argument has been raised that people couldn't engage 
in this sort of activity AND continue to review manuscripts for other 
journals; my counter is that this website would REPLACE each and 
every one of those print journals, at least insofar as taxonomic 
publication - there wouldn't BE any other manuscripts being published 
in any other journals that would require your reviews. If this 
website was the only place in the world that taxonomic publications 
were being submitted, reviewed, and published, then every taxonomist 
in the world would by default go there to submit their own works, and 
read those of their colleagues. I know that if I had a manuscript 
online and turned on my computer to see a message telling me "There 
are 7 new comments on your manuscript today" that I would certainly 
pay attention; another message saying "There are 3 new submissions in 
your target subjects, by authors X, Y, and Z, the second of which has 
specifically requested your review" would also get my attention. If 
each of those 7 comments took me only a few minutes to address, then 
it wouldn't be much of a disruption of my daily routine, nor would 
the 3 manuscripts, since I could look them over at my leisure, and 
make comments and corrections in a similarly casual fashion.

Which would be easier for you as a reviewer: (A) to receive 10 
manuscripts a year, received at random intervals, with deadlines by 
which you needed to read through the entirety of each, with only ONE 
opportunity to make ALL of your comments (many of which would be 
completely redundant with the comments from other reviewers), and NO 
opportunity to see any responses to your criticisms before it was in 
print, or (B) to have 10 manuscripts available online simultaneously, 
over a period of as long as a year or two, where you can edit them 
directly, and any corrections or improvements made by other reviewers 
would appear instantly (which is what a Wiki is all about), and you 
can take however little or as much time as you want to review any 
part of any manuscript, and get direct and timely feedback from the 
author(s) when you submit a question or criticism?

Using that kind of system, even the most time-stressed of us would, I 
believe, find the review process far more friendly and useful, both 
as a reviewer and as an author. It might also result in more 
completed works involving co-authorship, as a by-product of the 
collaborative nature of the review process (e.g., if author X submits 
a revisionary work describing new species in Genus A, and author Y 
turns out to also be revising that genus, and they have a few 
additional species that author A does not have, then the wiki process 
would allow them to compare notes and collaborate online, rather than 
each competing to see who can publish their revision first).

Hopefully, this is sounding a little less like a dream and more like 
something attainable?

Sincerely,
-- 

Doug Yanega        Dept. of Entomology         Entomology Research Museum
Univ. of California, Riverside, CA 92521-0314        skype: dyanega
phone: (951) 827-4315 (standard disclaimer: opinions are mine, not UCR's)
              http://cache.ucr.edu/~heraty/yanega.html 
   "There are some enterprises in which a careful disorderliness
         is the true method" - Herman Melville, Moby Dick, Chap. 82

_______________________________________________
Taxacom mailing list
Taxacom at mailman.nhm.ku.edu 
http://mailman.nhm.ku.edu/mailman/listinfo/taxacom