Peer Review for Journals: Evidence on Quality Control, Fairness, and Innovation

In the references below, the empirical studies that were used in this paper are designated by an 'E'. If the paper was based on an experiment or quasi-experiment, an 'X' follows the 'E.' If an author of this paper replied to my request for information about the coding, or if the paper was by the author of the current paper, the paper is designated by an 'ER,' or 'EXR' for experimental papers.

Abramowitz S I, Gomes B & Abramowitz C V (1975) Publish or politic: Referee bias in manuscript review. Journal of Applied Social Psychology 5: No. 3, 187-200. EX

Abrams P A (1991) The predictive ability of peer review of grant proposals: The case of ecology and the US National

Science Foundation. Social Studies of Science 21, 111-132. ER

Arkes H (1996) The persistence of management folklore. Interfaces 26, No. 4, 42-44.

Armstrong J S (1980) Unintelligible management research and academic prestige. Interfaces 10 (April), 80-86. EXR

Armstrong J S (1982) Barriers to scientific contributions: The author=s formula. The Behavioral and Brain Sciences 5,

197-199. ER

Armstrong J S (1985) Long-Range Forecasting. New York: John Wiley.

Armstrong J S (1996) Management folklore and management science: On portfolio planning, escalation bias, and such

(with commentaries). Interfaces 26: No. 4, 25-55.

Armstrong J S & Hubbard R (1991) Does the need for agreement among reviewers inhibit the publication of

controversial findings? Behavioral and Brain Sciences 14: (March), 136-137. ER

Atkinson D R, Furlong M J & Wampold B E (1982) Statistical significance, reviewer evaluations, and the scientific

process: Is there a statistically significant relationship? Journal of Counseling Psychology 29, No. 2, 189-194. EX

Bakanic V, McPhail C & Simon R J (1990) If at first you don=t succeed: Review procedures for revised and resubmitted manuscripts. American Sociologist 21, No 4, 373-391. E

Barber B (1961) Resistance by scientists to scientific discovery. Science 134, 596-602.

Begg C B & Berlin J A (1988) Publication bias: A problem in interpreting medical data. Journal of the Royal Statistical Society A 151, 419-463. E

Beyer J M, Chanove R G & Fox W B (1995) The review process and the fates of manuscripts submitted to AMJ. Academy of Management Journal 38, 1219-1260. ER

Blank R M (1991) The effects of double-blind versus single-blind reviewing: Experimental evidence from the American

Economic Review. American Economic Review 81, 1041-1067. EXR

Bradley J V (1981) Pernicious publication practices. Bulletin of the Psychonomic Society 18, 31-34. E

Broad W & Wade N (1982) Betrayers of the Truth. New York: Simon and Schuster.

Campanario J M (1995) On influential books and journal articles initially rejected because of negative referees=

evaluations. Science Communication 16 (March), 304-325. ER

Chalmers I (1990) Underreporting research is scientific misconduct. Journal of the American Medical Association 263, No. 10, 1405-1408.

Chalmers T C, Frank C S & Reitman D (1990) Minimizing the three stages of publication bias. Journal of the American

Medical Association 263, No. 10, 1392-1395. E

Christiansen D (1978) The perils of publishing. IEEE Spectrum 15, No. 5, 27.

Cho M K & Bero L A (1996) The quality of drug studies published in symposium proceedings.Annals of Internal

Medicine 124, No. 5, 485-489. E

Cicchetti D V (1991) The reliability of peer review for manuscript and grant submissions: A cross-disciplinary

investigation. Behavioral and Brain Sciences 14 (March), 119-186. E

Cohen J (1994) The earth is round (p < .05). American Psychologist 49, 997-1003.

Davidson R A (1986) Source of funding and outcome of clinical trials. Journal of General Internal Medicine 1, 155-158. ER

Dewald W G, Thursby J G & Anderson R G (1986) Replication in empirical economics: The Journal of Money, Credit,

and Banking project. American Economic Review 76, 587-603. E

Eichorn P & Yankauer A (1987) Do authors check their references? A survey of accuracy of references in three public health journals. American Journal of Public Health 77, 1011-1012. ER

Epstein W M (1990) Confirmational response bias among social work journals. Science, Technology, and Human Values

15, 9-38. EXR

Evans J T, Nadjari H I & Burchell S A (1990) Quotational and reference accuracy in surgical journals: A continuing peer review problem. Journal of the American Medical Association 263, No. 10, 1353-1354. E

Fagan W T (1990) To accept or reject: Peer review. Journal of Educational Thought 24, 103-113. ER

Fletcher R H & Fletcher S W (1997) Evidence for the effectiveness of peer review. Science and Engineering Ethics, 3, 35-50. EX

Franke R H, Edlund T W & Oster F (1990) The development of strategic management: Journal quality and article impact. Strategic Management Journal 11, 243-253. ER

Fuess S M (1996) On replication in business and economics research: The QJBE case. Quarterly Journal of Business and

Economics 35, No. 2, 3-13. E

Friedman P J (1990) Correcting the literature following fraudulent publication. Journal of the American Medical

Association 263, 1416-1419. ER

Gans J S & Shepherd G B (1994) How are the mighty fallen: Rejected classic articles by leading economists. Journal of

Economic Perspectives 8, No. 1, pp. 165-179. E

Garcia J (1981) Tilting at the paper mills of academe. American Psychologist 36, No. 2, 149-158.

Garfield E & Welljams-Dorof A (1990) The impact of fraudulent research on the scientific literature. Journal of the American Medical Association 263, No. 10, 1424-1426. E

Goodstein L & Brazis, K (1970) Credibility of psychologists: An empirical study. Psychological Reports 27, No. 3, 835-838. E

Gottfredson S D (1978) Evaluating psychological research reports: Dimensions, reliability, and correlates of quality

judgments. American Psychologist 33, 920-934. EX

Greenwald A G (1975) Consequences of prejudice against the null hypothesis. Psychological Bulletin 82, 1-20. E

Hargens L L (1990) Variation in journal peer review systems: possible causes and consequences. Journal of the

American Medical Association 263, 1348-1352.

Harnad S (1979) Creative disagreement. The Sciences 19, 18-20.

Hendrick C (1976) Editorial comment. Personality and Social Psychology Bulletin 2, No. 3, 207-208.

Hibbitts B J (1996a) Last writes? Re-assessing the law review in the age of cyberspace.

http://www.law.pitt.edu/hibbits/lastrev.htm; version 1.1, June 4, 1996; New York University Law Review 17, 615-

688.

Hibbitts B J (1996b) Yesterday once more: Skeptics, scribes and the demise of law reviews. Akron Law Review

(forthcoming).

Holub H W, Tappeiner G & Eberharter V (1991) The iron law of important articles. Southern Economic Journal 58,

317-328. ER

Horrobin D F (1982) A philosophically faulty concept which is proving disastrous for science. Behavioral and Brain

Sciences 5, No. 2, 217-218.

Horrobin D F (1990) The philosophical basis of peer review and the suppression of innovation, Journal of the American

Medical Association 263 (March 9), 1438-1441.

Horrobin D F (1996) Peer review of research grant application. Lancet (forthcoming).

Hubbard R & Armstrong J S (1992) Are null results becoming an endangered species in marketing? Marketing Letters 3,

127-136. ER

Hubbard R & Armstrong J S (1994) Replications and extensions in marketing: Rarely published but quite contrary. International Journal of Research in Marketing 11, 233-248. ER

Hubbard R & Vetter D E (1996) An empirical comparison of published replication research in accounting, economics,

finance, management and marketing. Journal of Business Research 35, 153-164. ER

Jauch L R & Wall J L (1989) What they do when they get your manuscript: A survey of Academy of Management

reviewer practices. Academy of Management Journal 32, 157-173. E

Juhasz S, Calvert E, Jackson T, Kronick D A & Shipton J (1975) Acceptance and rejection of manuscripts. IEEE

Transactions of Professional Communications PC18, 177-184. E

Kerr S, Tolliver J & Petree D (1972) Manuscript characteristics which influence acceptance for management and social science journals. Academy of Management Journal 20, No. 1, 132-141. ER

King D W, McDonald D D & Roderer N K (1981) Scientific Journals in the United States: Their Production, Use, and

Economics. Stroudsburg, Pa: Hutchison Ross. E

Koehler J J (1993) The influence of prior beliefs on scientific judgments of evidence quality. Organizational Behavior and Human Decision Processes 56, 28-55. EXR

Kuhn T S (1962) The Structure of Scientific Revolutions. Chicago: University of Chicago Press.

Kupfersmid J & Wonderly D M (1994) An Author=s Guide to Publishing Better Articles in Better Journals in the

Behavioral Sciences. Brandon, Vermont: Clinical Psychology Publishing Co. ER

Lau R R (1994) An analysis of the accuracy of ?trial heat@ polls during the 1992 presidential election. Public Opinion

Quarterly 58, 2-20.

Lindsey D (1978) The Scientific Publication System in Social Science. San Francisco: Jossey-Bass. E

Liversidge A (1989) PNAS publication of AIDS article spurs debate over peer review. The Scientist 3, No. 7, 4-5, 19.

Lock S & Smith J (1986) Peer review at work. Scholarly Publishing 17, No. 4, 303-316. E

Lock S & Smith J (1990) What do peer reviewers do? Journal of the American Medical Association 263, No. 10, 1341-

1343. E

Lloyd M E (1990) Gender factors in reviewer recommendations for manuscript publication. Journal of Applied Behavior Analysis 23, 539-543. EX

MacNealy M S, Speck B W & Clements N (1994) Publishing in technical communication journals from the successful

author's point of view. Technical communication 41, No. 2, 240-259. E

Mahoney M (1977) Publication prejudices: An experimental study of confirmatory bias in the peer review system.

Cognitive Therapy and Research 1, 161-175. EXR

Marsh H W & Ball S (1989) The peer review process used to evaluate manuscripts submitted to academic journals:

Interjudgmental reliability. Journal of Experimental Education 57, No. 2, 151-169. E

McCloskey D N & Ziliak S T (1996) The standard error of regressions. Journal of Economic Literature 34 (March), 97-

114. E

McNutt R A, Evans A T, Fletcher, R H & Fletcher S W (1990) The effects of blinding on the quality of peer review: A

randomized trial. Journal of the American Medical Association 263, No. 10, 1371-1376. EX

Munley P H, Sharkin B & Gelso C J (1988) Reviewer ratings and agreement on manuscripts reviewed for the Journal of

Counseling Psychology. Journal of Counseling Psychology 35, No. 2, 198-202. E

Murray G D (1988) The task of a statistical referee. British Journal of Surgery 75, 664-667. E

Needleman H L (1992) Salem comes to the National Institutes of Health: Notes from inside the crucible of scientific integrity. Pediatrics 90, No. 6, 977-981.

Newcombe R G (1987) Towards a reduction in publication bias. British Medical Journal 295 (12 September), 656-659.

Patterson S C & S K Smithey (1990) Monitoring scholarly journal publication in political science: The role of the APSR. PS: Political Science and Politics 23, 647-656. ER

Perlman D (1982) Reviewer ?bias@: Do Peters and Ceci protest too much? The Behavioral and Brain Sciences 5, 231-

232. E

Pfeifer M P & Snodgrass G L (1990) The continued use of retracted, invalid scientific literature. Journal of the American Medical Association 263, No. 10 (1990), 1420-1423. E

Peters D P & Ceci S J (1982) Peer-review practices of psychology journals: The fate of published articles, submitted again. The Behavioral and Brain Sciences 5, 187-195. EX

Pressman S (1994) Simultaneous multiple journal submissions: The case against. American Journal of Economics and Sociology 53, 316-333.

Rodman H & Mancini J A (1977) Errors, manuscripts, and equal treatment. Research in Higher Education 7, 369-374.

ER

Roediger, H L (1987) The role of journal editors in the scientific process. In D N Jackson and J P Rushton, Scientific Excellence. London: Sage Publications.

Rowney J A & Zenisek T J (1980) Manuscript characteristics influencing reviewers= decisions. Canadian Psychology 21,

17-21. ER

Rushton J P & Ankney C D (1996) Brain size and cognitive ability: Correlations with age, sex, social class, and race.

Psychonomic Bulletin and Review 3, No. 1, 21-36.

Salsburg D S (1985) The religion of statistics as practiced in medical journals. American Statistician 39, 220-223. E

Shea C (1996) Psychologists debate accuracy of <significance test. The Chronicle of Higher Education 42 (August 16), A12 & A17.

Simon R, Bakanic V & McPhail C (1986) Who complains to editors and what happens. Sociological Inquiry 56, 259-

271. ER

Smart R G (1964) The importance of negative results in psychological research. Canadian Psychologist 5, 225-232. E

Speck B W (1993) Publication Peer Review. Westport, Connecticut.

Spiegel-Rosing I (1977) Bibliometric and content analysis. Social Studies of Science 7, 97-113. E

Stamps A E III (1997) Advances in peer review research: An introduction. Science and Engineering Ethics (forthcoming)

ER

Sterling T D (1959) Publication decisions and their possible effects on inferences drawn from tests of significance - or vice versa. Journal of the American Statistical Association, 54, 30-34. ER

Sterling T D, Rosenbaum W L & Weinkam J J (1995) Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. American Statistician 49, 108-112. ER

Stewart W W & Feder N (1987) The integrity of the scientific literature. Nature 325, 207-214. ER

Szenberg M (1994) Disseminating scholarly output: The case for eliminating the exclusivity of journal submissions.

American Journal of Economics and Sociology 53, 303-315.

Taubes G (1994) Peer review in cyberspace. Science 266, 967.

Tetlock P E & Kim J I (1987) Accountability and judgment process in a personality prediction task. Journal of

Personality and Social Psychology 52, 700-709.

Wade N (1976) IQ and heredity: Suspicion of fraud beclouds classic experiment. Science 194, 916-919.

Weiss D J (1989) An experiment in publication: Advance publication review. Applied Psychological Measurement 13, 1-7.

Wilson J D (1978) Peer review and publication. Journal of Clinical Investigation 61, 1697-1701.

Yankauer A (1985) Peering at peer review. CBE Views 8, No. 2, 7-10. ER

Yankauer A (1990) Who are the peer reviewers and how much do they review? Journal of the American Medical

Association 263, 1338-1340. ER