Harnad, S. (2000) The Invisible Hand of Peer Review, Exploit Interactive, issue 5, April 2000
Electronics and Computer Science Department
SO17 1BJ United Kingdom
So it is in civic matters, and it is no different in the world of Learned
Inquiry. The "quis custodiet" problem among scholars has traditionally
been solved by means of a quality-control and certification [QC/C] system
called "peer review" (Harnad 1985): The work of specialists is submitted
to a qualified adjudicator, an editor, who in turn sends it to fellow-specialists,
referees, to seek their advice about whether the paper is potentially publishable,
and if so, what further work is required to make it acceptable. The paper
is not published until and unless the requisite revision can be and is
done to the satisfaction of the editor and referees.
Neither the editor nor the referees is infallible. Editors can err in the choice of specialists (indeed, it is well-known among editors that a deliberate bad choice of referees can always ensure that a paper is either accepted or rejected, as preferred); or editors can misinterpret or misapply referees' advice. The referees themselves can fail to be sufficiently expert, informed, conscientious or fair.
Nor are authors always conscientious in accepting the dictates of peer review. (It is likewise well-known among editors that virtually every paper is eventually published, somewhere (Lock 1985; Harnad 1986): There is a quality hierarchy among journals, based on the rigour of their peer review, all the way down to an unrefereed vanity press at the bottom. Persistent authors can work their way down until their paper finds its own level, not without considerable wasting of time and resources along the way, including the editorial office budgets of the journals and the freely given time of the referees, who might find themselves called upon more than once to review the same paper, sometimes unchanged, for several different journals.)
The system is not perfect, but it is what has vouchsafed us our refereed
journal literature to date, such as it is, and so far no one has demonstrated
any viable alternative to having experts judge the work of their peers,
let alone one that is at least as effective in maintaining the quality
of the literature as the present imperfect one is (Harnad 1982).
There is a way to test our intuitions about the merits of this sort of proposal a priori, using a specialist domain that is somewhat more urgent and immediate than abstract "learned inquiry"; if we are not prepared to generalise this intuitive test's verdict to scholarly/scientific research in general, we really need to ask ourselves how seriously we take the acquisition of knowledge: If someone near and dear to you were ill with a serious but potentially treatable disease, would you prefer to have them treated on the basis of the refereed medical literature or on the basis of an unfiltered free-for-all where the distinction between reliable expertise and ignorance, incompetence or charlatanism is left entirely to the reader, on a paper by paper basis?
A variant on this scenario is currently being tested by the British
Medical Journal <http://www.bmj.com/cgi/shtml/misc/peer/index.shtml>,
but instead of entrusting entirely to the reader the quality control function
performed by the referee in classical peer review, this variant, taking
a cue from some of the developments and goings-on on both the Internet
and Network TV chat-shows, plans to publicly post submitted papers unrefereed
on the Web and to invite any reader to submit a commentary; these commentaries
will then be used in lieu of referee reports as a basis for deciding on
Is this peer review? Well, it is not clear whether the self-appointed commentators will be qualified specialists (or how that is to be ascertained). The expert population in any given speciality is a scarce resource, already overharvested by classical peer review, so one wonders who would have the time or inclination to add journeyman commentary services to this load on their own initiative, particularly once it is no longer a rare novelty, and the entire raw, unpoliced literature is routinely appearing in this form first. Are those who have nothing more pressing to do with their time than this really the ones we want to trust to perform such a critical QC/C function for us all?
And is the remedy for the possibility of bias or incompetence in referee-selection
on the part of editors really to throw selectivity to the winds, and let
referees pick themselves? Considering all that hangs on being published
in refereed journals, it does not take much imagination to think of ways
authors could manipulate such a public-polling system to their own advantage,
human nature being what it is.
And is peer commentary (even if we can settle the vexed "peer" question) really peer review? Will I say publicly about someone who might be refereeing my next grant application or tenure review what I really think are the flaws of his latest raw manuscript? (Should we then be publishing our names alongside our votes in civic elections too, without fear or favour?) Will I put into a public commentary -- alongside who knows how many other such commentaries, to be put to who knows what use by who knows whom -- the time and effort that I would put into a referee report for an editor I know to be turning specifically to me and a few other specialists for our expertise on a specific paper?
If there is anyone on this planet who is in a position to attest to the functional difference between peer review and peer commentary (Harnad 1982, 1984), it is surely the author of the present article, who has been umpiring a peer-reviewed paper journal of Open Peer Commentary (Behavioral and Brain Sciences [BBS] <http://www.princeton.edu/~harnad/bbs.html>, published by Cambridge University Press) for over 2 decades (Harnad 1979), as well as a brave new online-only journal of Open Peer Commentary, likewise peer-reviewed (Psycoloquy, sponsored by the American Psychological Association, <http://www.princeton.edu/~harnad/psyc.html>), which entered its second decade with the millennium.
Both journals are rigorously refereed; only those papers that have successfully passed through the peer review filter go on to run the gauntlet of open peer commentary, an extremely powerful and important supplement to peer review, but certainly no substitute for it. Indeed, no one but the editor sees [or should have to see] the population of raw, unrefereed submissions, consisting of some manuscripts that are eventually destined to be revised and accepted after peer review, but also (with a journal like BBS, having a 75% rejection rate) many manuscripts not destined to appear in that particular journal at all. Referee reports, some written for my eyes only, all written for at most the author and fellow referees, are nothing like public commentaries for the eyes of the entire learned community, and vice versa. Nor do 75% of the submissions justify soliciting public commentary, or at least not commentary at the BBS level of the hierarchy.
It has been suggested that in fields such as Physics, where the rejection
rate is lower (perhaps in part because the authors are more disciplined
and realistic in their initial choice of target journal, rather than trying
their luck from the top down), the difference between the unrefereed preprint
literature and the refereed reprint literature may not be that great; hence
one is fairly safe using the unrefereed drafts, and perhaps the refereeing
could be jettisoned altogether.
Is this really evidence that peer review is not indispensable after
all? Hardly, for the "Invisible Hand" of peer review is still there, exerting
its civilising influence: Just about every paper deposited in Los Alamos
is also destined for a peer reviewed journal; the author knows it will
be answerable to the editors and referees. That certainly constrains how
it is written in the first place. Remove that invisible constraint -- let
the authors be answerable to no one but the general users of the Archive
(or even its self-appointed "commentators") -- and watch human nature take
its natural course, standards eroding as the Archive devolves toward the
canonical state of unconstrained postings: the free-for-all chat-groups
of Usenet <http://tile.net/news/listed.html>,
that Global Graffiti Board for Trivial Pursuit -- until someone re-invents
peer review and quality control.
Now it is no secret that I am a strong advocate of a free literature along the lines of Los Alamos (Okerson & O'Donnell 1995). How are we to reconcile the conservative things I've said about QC/C here with the radical things I've advocated elsewhere about public author archiving (Harnad 1998a, 1998b, 1999)?
The answer is very simple. The current cost of the refereed paper journal literature is paid for by Subscription, Site License and Pay-Per-View (S/L/P). Both the medium (paper) and the method of cost-recovery (S/L/P) share the feature that they block access to the refereed literature, whereas the authors, who contribute their papers for free, would infinitely prefer free, universal access to their work.
The optimal (and inevitable) solution is an online-only refereed journal
literature, which will be much less costly (less than 1/3 of the current
price per page) once it is paper-free <http://amsci-forum.amsci.org/archives/september98-forum.html>
and resides in open archives
<http://www.openarchives.org>but still not entirely cost-free, because the peer review (and editing) still needs to be paid for (Odlyzko 1998). If those residual QC/C costs are paid at the author-institution-end (not out of the author's pocket, of course, but out of instiutional publication funds redirected from 1/3 of the 3/3 annual institutional savings from serial S/L/P cancellations), the dividend will be that the papers are all accessible for free for all (via interoperable open archives such as CogPrints <http://cogprints.soton.ac.uk > -- integrated seamlessly into a single global "virtual" archive, mirrored worldwide, which will then have an unrefereed preprint sector and a refereed, published, reprint sector, tagged by journal name). Journal publishers will continue to provide and be paid for their QC/C while the public archive will serve as the "front end" for both journal submissions (tagged "unrefereed prepints") and published articles (tagged "refereed reprints [plus journal name, etc.]"<http://www.eprints.org/index.html>.
To distribute the load among referees more equitably (and perhaps also
to protect editors from themeselves), the journal editor can formally approach
a much larger population of selected, qualified experts about relevant
papers they are invited to referee if they have the time and inclination.
Referee reports can be emailed or deposited directly through a password-controlled
Web interface. Accepted final drafts can be edited and marked up online,
and the final draft can then be deposited in the public Archive for all,
superseding the preprint.
Referee reports can be revised, published and linked to the published article as commentaries if the referee wishes; so can author rebuttals. And further commentaries, both refereed and unrefereed, can be archived and linked to the published article, along with author responses. Nor is there any reason to rule out postpublication author updates and revisions of the original article -- 2nd and 3rd editions, both unrefereed and refereed. Learned Inquiry, as I have had occasion to write before (Harnad 1990) is a continuum; reports of its findings -- informal and formal, unrefereed and refereed -- are milestones, not gravestones; as such, they need only be reliably sign-posted. The discerning hitch-hiker in the PostGutenberg Galaxy can take care of the rest (Harnad 1991).
Overall, the dissemination of learned research, once we have attained
the optimal and inevitable state described here, will be substantially
accelerated, universally accessible, and incomparably more interactive
in the age of Scholarly Skywriting than it was in our own pedestrian, papyrocentric
one; Learned Inquiry itself -- and hence all of society --will be
the chief beneficiary.
Ginsparg, P. (1996) Winners and Losers in the Global research Village. Invited contribution, UNESCO Conference HQ, Paris, 19-23 Feb 1996.
Ginsparg, P. (1994) First Steps Towards Electronic Research Communication.
Computers in Physics. (August, American Institute of Physics). 8(4): 390-396.
Harnad, S. (1979) Creative disagreement. The Sciences 19: 18 - 20.
Harnad, S. (ed.) (1982) Peer commentary on peer review: A case study in scientific quality control, New York: Cambridge University Press.
Harnad, S. (1984) Commentaries, opinions and the growth of scientific knowledge. American Psychologist 39: 1497 - 1498.
Harnad, S. (1985) Rational disagreement in peer review. Science, Technology and Human Values 10: 55 - 62.
Harnad, S. (1986) Policing the Paper Chase. (Review of S. Lock, A difficult balance: Peer review in biomedical publication.) Nature 322: 24 - 5.
Harnad, S. (1990) Scholarly Skywriting and the Prepublication Continuum
of Scientific Inquiry. Psychological Science 1: 342 - 343
Harnad, S. (1991) Post-Gutenberg Galaxy: The Fourth Revolution in the
Means of Production of Knowledge. Public-Access Computer Systems Review
2 (1): 39 - 53
Harnad, S. (1996) Implementing Peer Review on the Net: Scientific Quality
Control in Scholarly Electronic Journals. In: Peek, R. & Newby, G.
(Eds.) Scholarly Publishing: The Electronic Frontier. Cambridge MA: MIT
Press. Pp. 103-118.
Harnad, S. (1997) Learned Inquiry and the Net: The Role of Peer Review, Peer Commentary and Copyright. Learned Publishing 11(4) 283-292. Short version appeared in 1997 in Antiquity 71: 1042-1048. Excerpts also appeared in the University of Toronto Bulletin: 51(6) P. 12. http://citd.scar.utoronto.ca/EPub/talks/Harnad_Snider.html
Harnad, S. (1998a) For Whom the Gate Tolls? Free the Online-Only Refereed Literature. American Scientist Forum. http://www.cogsci.soton.ac.uk/~harnad/amlet.html
Harnad, S. (1998b) On-Line Journals and Financial Fire-Walls. Nature 395(6698): 127-128. http://www.cogsci.soton.ac.uk/~harnad/nature.html
Harnad, S. (1999) Free at Last: The Future of Peer-Reviewed Journals.
D-Lib Magazine 5(12) December 1999
Hitchcock, S. Carr, L., Jiao, Z., Bergmark, D., Hall, W., Lagoze, C.
& Harnad, S. (2000) Developing services for open eprint archives: globalisation,
integration and the impact of links. Proceedings of the 5th ACM Conference
on Digital Libraries. San Antonio Texas June 2000.
Carr, L., Hitchcock, S., Hall, W. & Harnad, S. (2000) A usage based
analysis of CoRR [A commentary on "CoRR: a Computing Research Repository"
by Joseph Y. Halpern] ACM SIGDOC Journal of Computer Documentation. May
Lock, Stephen (1985) A difficult balance : editorial peer review in medicine London : Nuffield Provincial Hospitals Trust.
Odlyzko, A.M. (1998) The economics of electronic journals. In: Ekman R. and Quandt, R. (Eds) Technology and Scholarly Communication Univ. Calif. Press, 1998. http://www.research.att.com/~amo/doc/economics.journals.txt
Okerson A. & O'Donnell, J. (Eds.) (1995) Scholarly Journals at the Crossroads; A Subversive Proposal for Electronic Publishing. Washington, DC., Association of Research Libraries, June 1995. http://www.arl.org/scomm/subversive/index.html
Youngen G.K. (1998) Citation patterns to traditional and electronic
preprints in the published literature. College and Research Libraries 59(5):