creators_name: Armstrong, J. Scott type: journalp datestamp: 2006-10-05 lastmod: 2011-03-11 08:56:38 metadata_visibility: show title: Peer Review for Journals: Evidence on Quality Control, Fairness, and Innovation ispublished: pub subjects: behanal full_text_status: public abstract: I reviewed the published empirical evidence concerning journal peer review, which consisted of 68 papers, all but three published since 1975. Peer review improves quality, but its use to screen papers has met with limited success. Current procedures to assure quality and fairness seem to discourage scientific advancement, especially important innovations, because findings that conflict with current beliefs are often judged to have defects. Editors can use procedures to encourage the publication of papers with innovative findings such as invited papers, early-acceptance procedures, author nominations of reviewers, results-blind reviews, structured rating sheets, open peer review, and, in particular, electronic publication. Some journals are currently using these procedures. The basic principle behind the proposals is to change the decision from whether to publish a paper to how to publish it. date: 1997 date_type: published publication: Science and Engineering Ethics volume: 3 pagerange: 63-84 refereed: FALSE referencetext: In the references below, the empirical studies that were used in this paper are designated by an 'E'. If the paper was based on an experiment or quasi-experiment, an 'X' follows the 'E.' If an author of this paper replied to my request for information about the coding, or if the paper was by the author of the current paper, the paper is designated by an 'ER,' or 'EXR' for experimental papers. Abramowitz S I, Gomes B & Abramowitz C V (1975) Publish or politic: Referee bias in manuscript review. Journal of Applied Social Psychology 5: No. 3, 187-200. EX Abrams P A (1991) The predictive ability of peer review of grant proposals: The case of ecology and the US National Science Foundation. Social Studies of Science 21, 111-132. ER Arkes H (1996) The persistence of management folklore. Interfaces 26, No. 4, 42-44. Armstrong J S (1980) Unintelligible management research and academic prestige. Interfaces 10 (April), 80-86. EXR Armstrong J S (1982) Barriers to scientific contributions: The author=s formula. The Behavioral and Brain Sciences 5, 197-199. ER Armstrong J S (1985) Long-Range Forecasting. New York: John Wiley. Armstrong J S (1996) Management folklore and management science: On portfolio planning, escalation bias, and such (with commentaries). Interfaces 26: No. 4, 25-55. Armstrong J S & Hubbard R (1991) Does the need for agreement among reviewers inhibit the publication of controversial findings? Behavioral and Brain Sciences 14: (March), 136-137. ER Atkinson D R, Furlong M J & Wampold B E (1982) Statistical significance, reviewer evaluations, and the scientific process: Is there a statistically significant relationship? Journal of Counseling Psychology 29, No. 2, 189-194. EX Bakanic V, McPhail C & Simon R J (1990) If at first you don=t succeed: Review procedures for revised and resubmitted manuscripts. American Sociologist 21, No 4, 373-391. E Barber B (1961) Resistance by scientists to scientific discovery. Science 134, 596-602. Begg C B & Berlin J A (1988) Publication bias: A problem in interpreting medical data. Journal of the Royal Statistical Society A 151, 419-463. E Beyer J M, Chanove R G & Fox W B (1995) The review process and the fates of manuscripts submitted to AMJ. Academy of Management Journal 38, 1219-1260. ER Blank R M (1991) The effects of double-blind versus single-blind reviewing: Experimental evidence from the American Economic Review. American Economic Review 81, 1041-1067. EXR Bradley J V (1981) Pernicious publication practices. Bulletin of the Psychonomic Society 18, 31-34. E Broad W & Wade N (1982) Betrayers of the Truth. New York: Simon and Schuster. Campanario J M (1995) On influential books and journal articles initially rejected because of negative referees= evaluations. Science Communication 16 (March), 304-325. ER Chalmers I (1990) Underreporting research is scientific misconduct. Journal of the American Medical Association 263, No. 10, 1405-1408. Chalmers T C, Frank C S & Reitman D (1990) Minimizing the three stages of publication bias. Journal of the American Medical Association 263, No. 10, 1392-1395. E Christiansen D (1978) The perils of publishing. IEEE Spectrum 15, No. 5, 27. Cho M K & Bero L A (1996) The quality of drug studies published in symposium proceedings.Annals of Internal Medicine 124, No. 5, 485-489. E Cicchetti D V (1991) The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation. Behavioral and Brain Sciences 14 (March), 119-186. E Cohen J (1994) The earth is round (p < .05). American Psychologist 49, 997-1003. Davidson R A (1986) Source of funding and outcome of clinical trials. Journal of General Internal Medicine 1, 155-158. ER Dewald W G, Thursby J G & Anderson R G (1986) Replication in empirical economics: The Journal of Money, Credit, and Banking project. American Economic Review 76, 587-603. E Eichorn P & Yankauer A (1987) Do authors check their references? A survey of accuracy of references in three public health journals. American Journal of Public Health 77, 1011-1012. ER Epstein W M (1990) Confirmational response bias among social work journals. Science, Technology, and Human Values 15, 9-38. EXR Evans J T, Nadjari H I & Burchell S A (1990) Quotational and reference accuracy in surgical journals: A continuing peer review problem. Journal of the American Medical Association 263, No. 10, 1353-1354. E Fagan W T (1990) To accept or reject: Peer review. Journal of Educational Thought 24, 103-113. ER Fletcher R H & Fletcher S W (1997) Evidence for the effectiveness of peer review. Science and Engineering Ethics, 3, 35-50. EX Franke R H, Edlund T W & Oster F (1990) The development of strategic management: Journal quality and article impact. Strategic Management Journal 11, 243-253. ER Fuess S M (1996) On replication in business and economics research: The QJBE case. Quarterly Journal of Business and Economics 35, No. 2, 3-13. E Friedman P J (1990) Correcting the literature following fraudulent publication. Journal of the American Medical Association 263, 1416-1419. ER Gans J S & Shepherd G B (1994) How are the mighty fallen: Rejected classic articles by leading economists. Journal of Economic Perspectives 8, No. 1, pp. 165-179. E Garcia J (1981) Tilting at the paper mills of academe. American Psychologist 36, No. 2, 149-158. Garfield E & Welljams-Dorof A (1990) The impact of fraudulent research on the scientific literature. Journal of the American Medical Association 263, No. 10, 1424-1426. E Goodstein L & Brazis, K (1970) Credibility of psychologists: An empirical study. Psychological Reports 27, No. 3, 835-838. E Gottfredson S D (1978) Evaluating psychological research reports: Dimensions, reliability, and correlates of quality judgments. American Psychologist 33, 920-934. EX Greenwald A G (1975) Consequences of prejudice against the null hypothesis. Psychological Bulletin 82, 1-20. E Hargens L L (1990) Variation in journal peer review systems: possible causes and consequences. Journal of the American Medical Association 263, 1348-1352. Harnad S (1979) Creative disagreement. The Sciences 19, 18-20. Hendrick C (1976) Editorial comment. Personality and Social Psychology Bulletin 2, No. 3, 207-208. Hibbitts B J (1996a) Last writes? Re-assessing the law review in the age of cyberspace. http://www.law.pitt.edu/hibbits/lastrev.htm; version 1.1, June 4, 1996; New York University Law Review 17, 615- 688. Hibbitts B J (1996b) Yesterday once more: Skeptics, scribes and the demise of law reviews. Akron Law Review (forthcoming). Holub H W, Tappeiner G & Eberharter V (1991) The iron law of important articles. Southern Economic Journal 58, 317-328. ER Horrobin D F (1982) A philosophically faulty concept which is proving disastrous for science. Behavioral and Brain Sciences 5, No. 2, 217-218. Horrobin D F (1990) The philosophical basis of peer review and the suppression of innovation, Journal of the American Medical Association 263 (March 9), 1438-1441. Horrobin D F (1996) Peer review of research grant application. Lancet (forthcoming). Hubbard R & Armstrong J S (1992) Are null results becoming an endangered species in marketing? Marketing Letters 3, 127-136. ER Hubbard R & Armstrong J S (1994) Replications and extensions in marketing: Rarely published but quite contrary. International Journal of Research in Marketing 11, 233-248. ER Hubbard R & Vetter D E (1996) An empirical comparison of published replication research in accounting, economics, finance, management and marketing. Journal of Business Research 35, 153-164. ER Jauch L R & Wall J L (1989) What they do when they get your manuscript: A survey of Academy of Management reviewer practices. Academy of Management Journal 32, 157-173. E Juhasz S, Calvert E, Jackson T, Kronick D A & Shipton J (1975) Acceptance and rejection of manuscripts. IEEE Transactions of Professional Communications PC18, 177-184. E Kerr S, Tolliver J & Petree D (1972) Manuscript characteristics which influence acceptance for management and social science journals. Academy of Management Journal 20, No. 1, 132-141. ER King D W, McDonald D D & Roderer N K (1981) Scientific Journals in the United States: Their Production, Use, and Economics. Stroudsburg, Pa: Hutchison Ross. E Koehler J J (1993) The influence of prior beliefs on scientific judgments of evidence quality. Organizational Behavior and Human Decision Processes 56, 28-55. EXR Kuhn T S (1962) The Structure of Scientific Revolutions. Chicago: University of Chicago Press. Kupfersmid J & Wonderly D M (1994) An Author=s Guide to Publishing Better Articles in Better Journals in the Behavioral Sciences. Brandon, Vermont: Clinical Psychology Publishing Co. ER Lau R R (1994) An analysis of the accuracy of ?trial heat@ polls during the 1992 presidential election. Public Opinion Quarterly 58, 2-20. Lindsey D (1978) The Scientific Publication System in Social Science. San Francisco: Jossey-Bass. E Liversidge A (1989) PNAS publication of AIDS article spurs debate over peer review. The Scientist 3, No. 7, 4-5, 19. Lock S & Smith J (1986) Peer review at work. Scholarly Publishing 17, No. 4, 303-316. E Lock S & Smith J (1990) What do peer reviewers do? Journal of the American Medical Association 263, No. 10, 1341- 1343. E Lloyd M E (1990) Gender factors in reviewer recommendations for manuscript publication. Journal of Applied Behavior Analysis 23, 539-543. EX MacNealy M S, Speck B W & Clements N (1994) Publishing in technical communication journals from the successful author's point of view. Technical communication 41, No. 2, 240-259. E Mahoney M (1977) Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research 1, 161-175. EXR Marsh H W & Ball S (1989) The peer review process used to evaluate manuscripts submitted to academic journals: Interjudgmental reliability. Journal of Experimental Education 57, No. 2, 151-169. E McCloskey D N & Ziliak S T (1996) The standard error of regressions. Journal of Economic Literature 34 (March), 97- 114. E McNutt R A, Evans A T, Fletcher, R H & Fletcher S W (1990) The effects of blinding on the quality of peer review: A randomized trial. Journal of the American Medical Association 263, No. 10, 1371-1376. EX Munley P H, Sharkin B & Gelso C J (1988) Reviewer ratings and agreement on manuscripts reviewed for the Journal of Counseling Psychology. Journal of Counseling Psychology 35, No. 2, 198-202. E Murray G D (1988) The task of a statistical referee. British Journal of Surgery 75, 664-667. E Needleman H L (1992) Salem comes to the National Institutes of Health: Notes from inside the crucible of scientific integrity. Pediatrics 90, No. 6, 977-981. Newcombe R G (1987) Towards a reduction in publication bias. British Medical Journal 295 (12 September), 656-659. Patterson S C & S K Smithey (1990) Monitoring scholarly journal publication in political science: The role of the APSR. PS: Political Science and Politics 23, 647-656. ER Perlman D (1982) Reviewer ?bias@: Do Peters and Ceci protest too much? The Behavioral and Brain Sciences 5, 231- 232. E Pfeifer M P & Snodgrass G L (1990) The continued use of retracted, invalid scientific literature. Journal of the American Medical Association 263, No. 10 (1990), 1420-1423. E Peters D P & Ceci S J (1982) Peer-review practices of psychology journals: The fate of published articles, submitted again. The Behavioral and Brain Sciences 5, 187-195. EX Pressman S (1994) Simultaneous multiple journal submissions: The case against. American Journal of Economics and Sociology 53, 316-333. Rodman H & Mancini J A (1977) Errors, manuscripts, and equal treatment. Research in Higher Education 7, 369-374. ER Roediger, H L (1987) The role of journal editors in the scientific process. In D N Jackson and J P Rushton, Scientific Excellence. London: Sage Publications. Rowney J A & Zenisek T J (1980) Manuscript characteristics influencing reviewers= decisions. Canadian Psychology 21, 17-21. ER Rushton J P & Ankney C D (1996) Brain size and cognitive ability: Correlations with age, sex, social class, and race. Psychonomic Bulletin and Review 3, No. 1, 21-36. Salsburg D S (1985) The religion of statistics as practiced in medical journals. American Statistician 39, 220-223. E Shea C (1996) Psychologists debate accuracy of