[Taxacom] Open review as a wiki

[Doug Yanega]

Neal Evenhuis asked:

>At 10:29 AM -0700 3/31/08, Doug Yanega wrote:
>>the model some of us are advocating
>>would be OPEN review - real-time, online, non-anonymous - in the
>>fashion of a Wiki. When all the criticisms of a submitted work have
>>been dealt with (by accommodating the valid criticisms and dismissing
>>the inappropriate ones)
>
>This is an admirable dream. I've been dreaming this too but being an 
>editor for over 30 years I've come up with many questions relating 
>to problems that ultimately need the human factor. If you've got 
>some answers, let me/us know.
>
>1. How would it be accomplished to take in the vagaries of 
>reviewers' habits including not responding in a timely manner?

How do various journals presently accommodate? For most, they solicit 
reviews, ask for a commitment to a deadline, and therefore skip 
potential reviewers who can't commit. Most print journals probably 
have only a slightly longer turnaround time than Zootaxa, where 
(unless I'm mistaken) the major time-saving step is the vastly 
shorter interval between approval and publication. However, bear in 
mind that  traditional reviews such as these only see a manuscript 
when the author(s) believe it is absolutely ready to publish!

Imagine an author who takes 4 years to get a manuscript ready for 
submission - if it takes another 4 months for it to get reviewed, 
revised, and printed, then that's great. Now, imagine that same 
author puts the first draft online for review right at the start; 
with feedback occurring throughout the process, and the streamlining 
influence of having a template to work from, it would very likely NOT 
take four years to finish it - maybe it would take only two years, 
let's say - in which case a statement like "Oh my god, we can't 
possibly tolerate a review process that takes two years!" is 
comparing apples to oranges (in this case, the manuscript got into 
print FASTER as a result of online review). As I said earlier, the 
submission of a manuscript would be part of the process of making 
specimen requests - it is certainly not uncommon for a study to take 
years to complete, if the amount of material being examined is 
substantial (for an entomologist, it would not be unusual to have to 
look over thousands of specimens from dozens of institutions). If 
reviewers can make comments during the time the author is examining 
specimens, then that leaves a VERY long time window, and even the 
laziest reviewer can't say that a year or more is not enough time to 
find an opportunity to make comments.

Remember also that the open review process will involve potentially 
every conceivable qualified reviewer, in which case an additional 
question should be asked: "Which is better, (A) a paper which goes 
from initial submission (as a finished work) to print in two months 
but is seen by only three reviewers, and is printed with errors and 
problems which a larger group of reviewers could have detected and 
fixed BEFORE it was printed, or (B) a paper which goes from initial 
submission (as a draft) to print in a year but which has been 
scrutinized by dozens of reviewers, including all of the pertinent 
authorities?" Just ask yourself how many times you've seen something 
incorrect or inappropriate come out in print that you KNOW you could 
have prevented had you been involved in the review. Under this 
system, everyone would *have* that opportunity, and everyone would 
benefit!

I would argue that many of the *nastiest* problems facing the 
taxonomic community can be traced to the lack of a universal standard 
of review; I believe firmly that making the review process as lengthy 
and exhaustive as possible is a desirable goal, with benefits that 
far outweigh any perception of encumbrance. High-quality work that 
meets the highest standards will, by its nature, pass through the 
system quickly and easily; work of lower quality will be held up only 
just as long as is necessary to bring it up to the higher standards 
desired.

>2. What happens when there may only be one specialist who could give 
>a good review and he/she is out of email contact for four weeks 
>while in the field?

Four weeks is nothing when a manuscript is taking shape over a year 
or two. Surely, some time during the development of a work there will 
be an opportunity for everyone with an interest to make their 
opinions known to the author(s), other reviewers, and referee.

>3. How are time limits to review set (three weeks, a month?)-- or 
>does the manuscript languish online for months or years on end in 
>hopes "someone" will notice it? The web is passive after all.
>4. Will there be some sort of registration of interested parties so 
>that they can get email automatic notices when their speciality 
>taxon group comes up for review?
>5. How do you know that everyone that is a specialist in a 
>particular group knows a manuscript is open for review?

These three are linked: there *would* be registration of interested 
parties, with automatic notification. There would also be a 
backtracking feature that would indicate who has not seen their 
notifications yet. Each incoming manuscript would have an impartial 
referee assigned to it, basically making sure the discussion stayed 
civil, that as many of the registered interested parties had made 
comments as seemed practical, and there would be a checklist of 
criteria that determined when something was ready to be accepted; 
most of these would be objectively-defined content (type deposition, 
etymology, non-homonymy, diagnoses, images/illustrations, etc.) that 
are BUILT INTO the template (so anything left blank would be 
glaringly obvious), but the crucial subjective decision would be "Are 
there any criticisms which have not been adequately addressed or 
refuted?". Ultimately, in such a system, I believe that the only 
manuscripts that would "languish" are those for which the author(s) 
failed to fill out the template completely in a timely manner, or 
could not (or would not) deal with criticisms adequately - remember, 
again, that the manuscripts would be online for a long, long time. If 
an author tries to conceal ongoing projects and refuses to submit 
them for review until they are "ready", then *those* authors may find 
the resulting review phase to be onerous, but only because they did 
not utilize the system in the manner it was intended.

>6. Bookmarking the "Open Review" site as one's web browser home page 
>might solve some of these problems but in reality, someone who sets 
>their web browser home page to default to news of a bunch of 
>taxonomy manuscripts available for review online would be a VERY 
>weird person who doesn't have much of a life. Any solution to this?

Many times the argument has been raised that people couldn't engage 
in this sort of activity AND continue to review manuscripts for other 
journals; my counter is that this website would REPLACE each and 
every one of those print journals, at least insofar as taxonomic 
publication - there wouldn't BE any other manuscripts being published 
in any other journals that would require your reviews. If this 
website was the only place in the world that taxonomic publications 
were being submitted, reviewed, and published, then every taxonomist 
in the world would by default go there to submit their own works, and 
read those of their colleagues. I know that if I had a manuscript 
online and turned on my computer to see a message telling me "There 
are 7 new comments on your manuscript today" that I would certainly 
pay attention; another message saying "There are 3 new submissions in 
your target subjects, by authors X, Y, and Z, the second of which has 
specifically requested your review" would also get my attention. If 
each of those 7 comments took me only a few minutes to address, then 
it wouldn't be much of a disruption of my daily routine, nor would 
the 3 manuscripts, since I could look them over at my leisure, and 
make comments and corrections in a similarly casual fashion.

Which would be easier for you as a reviewer: (A) to receive 10 
manuscripts a year, received at random intervals, with deadlines by 
which you needed to read through the entirety of each, with only ONE 
opportunity to make ALL of your comments (many of which would be 
completely redundant with the comments from other reviewers), and NO 
opportunity to see any responses to your criticisms before it was in 
print, or (B) to have 10 manuscripts available online simultaneously, 
over a period of as long as a year or two, where you can edit them 
directly, and any corrections or improvements made by other reviewers 
would appear instantly (which is what a Wiki is all about), and you 
can take however little or as much time as you want to review any 
part of any manuscript, and get direct and timely feedback from the 
author(s) when you submit a question or criticism?

Using that kind of system, even the most time-stressed of us would, I 
believe, find the review process far more friendly and useful, both 
as a reviewer and as an author. It might also result in more 
completed works involving co-authorship, as a by-product of the 
collaborative nature of the review process (e.g., if author X submits 
a revisionary work describing new species in Genus A, and author Y 
turns out to also be revising that genus, and they have a few 
additional species that author A does not have, then the wiki process 
would allow them to compare notes and collaborate online, rather than 
each competing to see who can publish their revision first).

Hopefully, this is sounding a little less like a dream and more like 
something attainable?

Sincerely,
-- 

Doug Yanega        Dept. of Entomology         Entomology Research Museum
Univ. of California, Riverside, CA 92521-0314        skype: dyanega
phone: (951) 827-4315 (standard disclaimer: opinions are mine, not UCR's)
              http://cache.ucr.edu/~heraty/yanega.html
   "There are some enterprises in which a careful disorderliness
         is the true method" - Herman Melville, Moby Dick, Chap. 82