creators_name: Armstrong, J. Scott type: journalp datestamp: 2006-09-25 lastmod: 2011-03-11 08:56:37 metadata_visibility: show title: Discovery and Communication of Important Marketing Findings: Evidence and Proposals ispublished: pub subjects: behanal full_text_status: public abstract: My review of empirical research on scientific publication led to the following conclusions. Three criteria are useful for identifying whether findings are important: replication, validity, and usefulness. A fourth criterion, surprise, applies in some situations. Based on these criteria, important findings resulting from academic research in marketing seem to be rare. To a large extent, this rarity is due to a reward system that is built around subjective peer review. Rather than using peer review as a secret screening process, using an open process likely will improve papers and inform readers. Researchers, journals, business schools, funding agencies, and professional organizations can all contribute to improving the process. For example, researchers should do directed research on papers that contribute to principles. Journals should invite papers that contribute to principles. Business school administrators should reward researchers who make important findings. Funding agencies should base decisions on researchers' prior success in making important findings, and professional organizations should maintain web sites that describe what is known about principles and what research is needed on principles. date: 2002 date_type: published publication: Journal of Business Research volume: 56 publisher: Elsevier Science Inc. pagerange: 69-84 refereed: FALSE referencetext: Abramowitz, S. I., B. Gomes, and C. V. Abramowitz (1975), "Publish or politic: Referee bias in manuscript review," Journal of Applied Social Psychology, 5, No. 3, 187-200. Abrami, P. C., L. Leventhal, and R. P. Perry (1982), “Educational seduction,” Review of Educational research, 52, 446-464. Abrams, P. A. (1991), “The predictive ability of peer review of grant proposals: The case of ecology and the U.S. National Science Foundation,” Social Studies of Science, 21, 111-132. AMA Task Force on the Development of Marketing Thought (1988), “Developing, disseminating, and utilizing marketing knowledge,” Journal of Marketing, 52, 1-25. Anderson, L. M. (1994), “Marketing science: Where’s the beef?” Business Horizons, (Jan-Feb), 8-16. Armstrong, J. S. (1980a), “Unintelligible management research and academic prestige,” Interfaces, 10 (April), 80-86. [available in full text at jscottarmstrong.com] Armstrong, J. S. (1980b), “Advocacy as a scientific strategy: The Mitroff myth,” Academy of Management Review, 5, 509-511. [available in full text at jscottarmstrong.com] Armstrong, J. S. (1982), “Barriers to scientific contributions: The author’s formula,” Behavioral and Brain Sciences, 5, (June), 197-199. [available in full text at jscottarmstrong.com] Armstrong, J. S. (1991), “Prediction of consumer behavior by experts and novices,” Journal of Consumer Research,” 18, 251-256. [available in full text at jscottarmstrong.com] Armstrong, J. S. (1995), “The devil’s advocate responds to an MBA student’s claim that research harms learning,” Journal of Marketing, 59, 101-106. [available in full text at jscottarmstrong.com] Armstrong, J. S. (1996), “Management folklore and management science: On portfolio planning, escalation bias and such,” Interfaces, 26 (July-August), 25-55 (includes commentary). [available in full text at jscottarmstrong.com] Armstrong, J. S. (1997), “Peer review for journals: Evidence on quality control, fairness, and innovation,” Science and Engineering Ethics, 3, 63-84. [available in full text at jscottarmstrong.com] Armstrong, J. S. and R. Brodie (1994), “Effects of portfolio planning methods on decision making: Experimental results,” International Journal of Research in Marketing, 11, 73-84. [available in full text at jscottarmstrong.com] Armstrong, J. S., R. Brodie and A. G. Parsons (2001), “Hypotheses in marketing science: Literature review and publication audit,” Marketing Letters, 12, 171-187. [available in full text at jscottarmstrong.com] Armstrong, J. S. and F. Collopy (1996), “Competitor orientation: Effects of objectives and information on managerial decisions and profitability,” Journal of Marketing Research, 33, 188-199. [available in full text at jscottarmstrong.com] Armstrong, J. S. and R. Hubbard (1991), “Does the need for agreement among reviewers inhibit the publication of controversial findings?” Behavioral and Brain Sciences, 14, 136-137.[available in full text at jscottarmstrong.com] Armstrong, J. S. and R. Schultz (1993), “Principles involving marketing policies: An empirical assessment,” Marketing Letters, 4, 253-265. [available in full text at jscottarmstrong.com] Armstrong, J. S. and T. Sperry (1994), “Business school prestige: Research versus teaching,” Interfaces, 24 (March-April), 13-22. [available in full text at jscottarmstrong.com] Atkinson, D. R., M. J. Furlong and B. E. Wampold (1982), "Statistical significance, reviewer evaluations, and the scientific process: Is there a (statistically) significant relationship?" Journal of Counseling Psychology, 29, 189-194. Bakanic, V., C. McPhail and R. J. Simon (1990), “If at first you don’t succeed: Reviewer procedures for revised and resubmitted manuscripts,” American Sociologist, 21, 373-391. Barber, B. (1961), “Resistance by scientists to scientific discovery,” Science, 134, 596-602. Begg, C. B. and J. A. Berlin (1988), "Publication bias: A problem in interpreting medical data," Journal of the Royal Statistical Society A, 151, 419-463. Beyer, J. M. (1978), "Editorial policies and practices among leading journals in four scientific fields," Sociological Quarterly, 19, 68-88. Bloom, P. N. (1987), Knowledge Development in Marketing. Lexington, MA: Lexington Books. Boland, R. J., Jr,. J. Singh, P. Salipante, J. D. Aram, S. Y. Fay and P. Kanawattanachai (2001), “Knowledge representations and knowledge transfer,” Academy of Management Journal, 44, 393-417. Borokowski, S. C. and M. J. Welsh (2000), “Ethical practice in the accounting publishing process: Contrasting opinions of authors and editors,” Journal of Business Ethics, 25, 15-31. Bradley, J. V. (1981), “Pernicious publication practices,” Bulletin of the Psychonomic Society,18, 31-34. Campanario, J. M. (1996), “The competition for journal space among referees, editors, and other authors and its influence on journals’ impact factors,” Journal of the American Society for Information Science, 47, 184-192. Campanario, J. M. (1998a), “Peer review for journals as it stands today – Part 1,” Science Communications, 19, 181-211. Campanario, J. M. (1998b), “Peer review for journals as it stands today – Part 2,” Science Communications, 19, 277-306. Chamberlin, T. C. (1890), “The method of multiple working hypotheses,” reprinted in Science, 148 (1965), 754-759. Cho, M. K. and L. A. Bero (1996), “The quality of drug studies published in symposium proceedings,” Annals of Internal Medicine, 124, 485-489. Chubin, L. D. and E. J. Hackett, Peerless Science: Peer Review and U. S. Science Policy. Albany NY: State University of New York Press. Cicchetti, D. V. (1991), “The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation,” Behavioral and Brain Sciences, 14 (March), 119-135. Cicchetti, D. V. (1997), "Referees, editors, and publication practices: Improving the reliability and usefulness of the peer review system," Science and Engineering Ethics, 3, 51-62. Cierpicki, S., M. Wright, and B. Sharp (2000) "Managers' knowledge of marketing principles: The case of new product development," Journal of Empirical Generalizations in Marketing Science, 5, 771-790. In full text at http://www.empgens.com/Pubs/jems/JEMS5_3.pdf Cohen, J. (1994), “The earth is round (p < .05),” American Psychologist, 49, 997-1003. Cole, J. R. and S. Cole (1972), “The Ortega hypothesis,” Science, 178 (27 October), 368-374. Dakin, S. & J. Armstrong (1989), "Predicting job performance: A comparison of expert opinion and research findings," International Journal of Forecasting, 5, 187-194. Davidson, R. A. (1986), “Source of funding and outcome of clinical trials,” Journal of Internal Medicine, 1, No. 3, 155-158. Eichorn, P. and A. Yankauer (1987), “Do authors check their references? A survey of accuracy of references in three public health journals,” American Journal of Public Health 77, 1011-1012. Epstein, W. M. (1990), “Confirmational response bias among social work journals,” Science, Technology, and Human Values, 15, 9-38. Evans, J. T., H. I. Nadjari and S. A. Burchell (1990), “ Quotational and reference accuracy in surgical journals: A continuing peer review problem,” Journal of the American Medical Association, 263, No. 10, 1353-1354. Faunce, G. J. and R. F. S. Job (2001) “The accuracy of reference lists in five experimental psychology journals,” American Psychologist, 56, 829-830. Feldman, K. A. (1987), “Research productivity and scholarly accomplishment of college teachers as related to their instructional effectiveness,” Research in Higher Education, 26, 227-298. Fletcher, R. H. and S. W. Fletcher (1997), “Evidence for the effectiveness of peer review,” Science and Engineering Ethics, 3, 35-50. Fox, K. J. and R. Milbourne (1999), “What determines research output of academic economists?” The Economic Record, 75, 257-267. Franke, R. H., T. W. Edlund, and F. Oster (1990), “The development of strategic management: Journal quality and article impact,” Strategic Management Journal, 11, 243-253. Friedman, M. (1992) “Correspondence,” Journal of Economic Perspectives, 4 (8), 199-200. Friedman, P. J. (1990), “Correcting the literature following fraudulent publication,” Journal of the American Medical Association, 263, No. 10, 1416-1419. Fuess, S. M. (1996), “On replication in business and economics research: The QJBE case,” Quarterly Journal of Business and Economics, 35 (2), 3-13. Gage, N. L. (1991), "The obviousness of social and educational research results," Educational Researcher, 20 (January), 10-16. Gans, J. S. and G. B. Shepard (1994), “How are the mighty fallen: Rejected classic articles by leading economists,” Journal of Economic Perspectives, 8, 165-179. Goodstein, L. D. and K. L. Brazis (1970), "Credibility of psychologists: An empirical study," Psychological Reports, 27 (1970), 835-838. Gordon, M. E., L. S. Kleiman and C. A. Hanie (1978), “Industrial-organizational psychology: Open thy ears O house of Israel,” American Psychologist, 33, 893-905. Gordon, G. and S. Marquis (1966), “Freedom, visibility of consequences and scientific innovation,” American Journal of Sociology, 72, 195-202. Gottfredson, S. D. (1978), “Evaluating psychological research reports: Dimensions, reliability, and correlates of quality judgments,” American Psychologist, 33, 920-934. Greenwald, A. G. (1975), "Consequences of prejudice against the null hypothesis," Psychological Bulletin, 82, 1-20. Hancock, T., J. Lane, R. Ray and D. Glennon (1992), “Factors influencing academic research productivity: A survey of management scientists,” (with commentary), Interfaces, 22 (September), 26-38. Harzing, A. (2002), “Are our referencing errors undermining our scholarship and credibility? The case of expatriate failure rates,” Journal of Organizational Behavior 23, 127-148. Hattie, J. and H. W. Marsh (1996), "The relationship between research and teaching: A metaanalysis," Review of Educational Research, 66, 507-542. Helgesen, T. (1994), “Advertising awards and advertising agency performance criteria,” Journal of Advertising Research, 34 (July-August), 43-53. Horrobin D. F. (1990), "The philosophical basis of peer review and the suppression of innovation," Journal of the American Medical Association, 263, 1438-1441. Hubbard, R. and J. S. Armstrong (1992), “Are null results becoming an endangered species in marketing?” Marketing Letters, 3, 127-136. [available in full text at jscottarmstrong.com] Hubbard, R. and J. S. Armstrong (1994), “Replications and extensions in marketing: Rarely published but quite contrary,” International Journal of Research in Marketing, 11, 233-248. [available in full text at jscottarmstrong.com] Hubbard, R. and D. E. Vetter (1996), “An empirical comparison of published replication research in accounting, economics, finance, management, and marketing,” Journal of Business Research, 35, 153-164. Hunt, S. D., L. B. Chonko and V. R. Wood (1986), “Marketing education and marketing success: Are they related?” Journal of Marketing Education, 6 (Summer), 2-13. Jauch, L. R. and J. L. Wall (1989), “What they do when they get your manuscript: A survey of Academy of Management reviewer practices,” Academy of Management Journal, 32, 157-173. Juhasz, S., E. Calvert, T. Jackson, D. A. Kronick and J. Shipman (1975), "Acceptance and rejection of manuscripts," IEEE Transactions of Professional Communications, PC, 18 (3), 177-184. Kealey, T. (1996), The Economic Laws of Scientific Research. London: MacMillan Press. Kerr, S., J. Tolliver and D. Petree (1977), “Manuscript characteristics which influence acceptance for management and social science journals,” Academy of Management Journal, 20, 132-141. King, D. W., D. D. McDonald and N. K. Roderer (1981), Scientific Journals in the United States: Their Production, Use, and Economics. Stroudsburg, Pa: Hutchison Ross. Kirkpatrick, S. A. and E. A. Locke (1992), “The development of measures of faculty scholarship,” Group and Organizational Management,” 17, No. 1, (March), 5-23. Koehler, J. J. (1993), "The influence of prior beliefs on scientific judgments of evidence quality," Organizational Behavior and Human Decision Processes, 56, 28-55. Kuhn, T. S. (1962), The Structure of Scientific Revolutions. Chicago: University of Chicago Press. Laband, D. N. and M. J. Piette (1994), “Favoritism versus search for good papers: Empirical evidence regarding the behavior of journal editors,” Journal of Political Economy, 102, 194-203. Lazarsfeld, P. F. (1949), “The American Soldier - An expository review," Public Opinion Quarterly, 13, 377-404. Lee, J. A. (1980), The Gold and the Garbage in Management Theories and Prescriptions. Athens, Ohio: Ohio University Press. Leone, R. P. and R. L. Schultz (1980), “A study of arketing generalizations,” Journal of Marketing, 44 (Winter), 10-18. Lindsey, D. (1978), The Scientific Publication System in Social Science. San Francisco: Jossey-Bass. Linsky, A. S. (1975), “Stimulating responses to mailed questionnaires: A review,” Public Opinion Quarterly, 39, 82-101. Lock, S. and J. Smith (1990)"What do peer reviewers do?" Journal of the American Medical Association, 263, 1341-1343. Luthans, F., R. M. Hodgetts and S. A. Rosenkrantz (1988), Real Managers. Cambridge, MA: Ballinger Publishing. Lynn, M. (1991), "Scarcity effects on value: A quantitative review of the commodity theory literature," Psychology and Marketing, 8, 43-57. MacNealy M. S., B. W. Speck and N. Clements (1994), “Publishing in technical communication journals from the successful author’s point of view,” Technical Communication, 41, No. 2, 240-259. Madden, C. S., L. S. Franz and R. A. Mittlestaedt (1979), “The replicability of research in marketing: Reported content and author cooperation,” in O. C. Ferrell, S. W. Brown and C. W. Lamb (eds.), Conceptual and Theoretical Developments in Marketing, pp. 77-87. Chicago: American Marketing Association. Mahoney, M. J. (1977). “Publication prejudices: An experimental study of confirmatory bias in the peer review system,” Cognitive Therapy and Research, 1, 161-175. Marsh, H. W. and S. Ball (1989), “The peer review process used to evaluate manuscripts submitted to academic journals: Interjudgmental reliability,” Journal of Experimental Education, 57, No. 2, 151-169. Martinko, M. J., C. R. Campbell and S. C. Douglas (2000), “Bias in the social science publication process: Are there exceptions?” Journal of Social Behavior and personality, 15 (1), 1-18. McClosky, D. N. and T. Ziliak (1996), “The standard error of regressions,” Journal of Economic Literature, 34, 97-114. Metoyer-Duran, C. (1993), “The readability of published, accepted, and rejected papers appearing in College & Research Libraries,” College & Research Libraries, (November), 517-526. Miner, J. B. (1984), “The validity and usefulness of theories in an emerging organizational science,” Academy of Management Review, 9, 296-306. Mischel, W. (1981), “Metacognition and the rules of delay,” in J. H. Flavell and L. Ross (eds.), Social Cognition Development. Cambridge, England: Cambridge University Press. Mitroff, I. (1972), “The myth of objectivity or why science needs a new psychology of science,” Management Science, 18, B613-B618. Munley, P. H., B. Sharkin and C. J. Gelso (1988) “Reviewer ratings and agreement on manuscripts reviewed for the Journal of Counseling Psychology,” Journal of Counseling Psychology, 35, 198-202. Murray, G. D. (1988), “The task of a statistical referee,” British Journal of Surgery 75 (7), 664- 667. Nagle, T. T. and R. K. Holden (1995), The Strategy and Tactics of Pricing. Upper Saddle River, NJ: Prentice Hall. Neuliep, J. W. and R. Crandall (1990), “Editorial bias against replication research,” Journal of Social Behavior and Personality, 5, 85-90. Oakes, M. (1986), Statistical Inference: A Commentary for the Social and Behavioral Sciences. New York: John Wiley. Peters, D. P. and S. J. Ceci (1982), “Peer-review practices of psychological journals: The fate of published articles, submitted again,” The Behavioral and Brain Sciences, 5, 187-195. Reid, L. N., H. J. Rotfeld and R. D. Wimmer (1982), “How researchers respond to replication requests,” Journal of Consumer Research, 9, 216-218. Rodman, H. and J. A. Mancini (1977), “Errors, manuscripts, and equal treatment,” Research in Higher Education, 7, 369-374. Rosenthal, R. and R. L. Rosnow (1984), Essentials of Behavioral Research: Methods and Data Analysis. New York: McGraw-Hill. Rossiter J. R. (2001), “What is marketing knowledge,” Marketing Theory, 1, 9-26. Rossiter, J. R. and L. Percy (1997), Advertising Communications and Promotion Management. New York: McGraw-Hill. Rothwell, P. M. and C. N. Martyn (2000), “Reproducibility of peer review in clinical neuroscience: Is agreement between reviewers any greater than would be expected by chance alone?” Brain, 123, 1964-1969. Rowney, J. A. and T. J. Zenisek (1980), “Manuscript characteristics influencing reviewers’ decisions,” Canadian Psychology, 21, 17-21. Salsburg, D. S. (1985), "The religion of statistics as practiced in medical journals," American Statistician, 39, 220-223. Scandura, T. and E. A. Williams (2000), "Research methodology in management: Current practices, trends, and implications for future research," Academy of Management Journal, 43, 1248-1264. Sherrell, D. L., J. F. Hair, Jr. and M. Griffin (1989), “Marketing academicians’ perceptions of ethical research and publishing behavior,” Journal of the Academy of Marketing Science, 17, 315-324. Sherwin, C. W. and R. S. Isenson (1967), “Project hindsight,” Science, 186, 1571-1577. Simon, R., V. Bakanic and C. McPhail (1986), “Who complains to the editors and what happens?” Sociological Inquiry, 56, 259-271. Singh, S. N. and R. F. Bush (1998), “Research burnout in tenured marketing professors: An empirical investigation,” Journal of Marketing Education, 20 (1), 4-15 Slovic, P. and B. Fischhoff (1977), “On the psychology of experimental surprises,” Journal of Experimental Psychology: Human Perception and Performance, 3, 544-551. Smart, R. G. (1964), "The importance of negative results in psychological research," Canadian Psychologist, 5, 225-232. Smith, K. and D. N. Laband (1995), “The role of editor’s professional connections in determining which papers get published: Evidence from accounting research journals,” Accounting Perspectives, 1, 21-30. Spiegel-Rosing, I. (1977), “Bibliometric and content analysis,” Social Studies of Science, 7, 97-113. Sterling, T. D. (1959), “Publication decisions and their possible effects on inferences drawn from tests of significance – or vice versa,” Journal of the American Statistical Association, 54, 30-34. Sterling T. D, Rosenbaum W. L. & J. J. Weinkam (1995), “Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa,” American Statistician, 49, 108-112. Stewart, W. W. and N. Feder (1987), “The integrity of the scientific literature,” Nature, 325, 207-214. Street, M. D., D. P. Bozeman and J. M. Whitfield (1998), "Author perceptions of positive and negative behaviors in the manuscript review process," Journal of Social Behavior and Personality, 13 (1), 1-22. Sutton R. I. and A. Rafaeli (1988), “Untangling the relationship between displayed emotions and organizational sales: The case of convenience stores,” Academy of Management Journal, 31, 461-487. Tellis, G. J. (1988), "The price elasticity of selective demand: A meta-analysis of econometric models of sales," Journal of Marketing Research, 25, 331-341. Trieschmann, J. S., A. R. Dennis, G. B. Northcraft and A. W. Niemi (2000), “Serving multiple constituencies in business schools: M.B.A. program versus research performance,” Academy of Management Journal, 43, 1130-1141. Van Fleet, D. D., A. McWilliams and D. S. Siegel (2000), “A theoretical and empirical analysis of journal rankings: The case of formal lists,” Journal of Management, 26, 839-861. Wells, W. D. (1993). “Discovery-oriented consumer research,” Journal of Consumer Research, 19, 489-504. Weller, A. (1990), “Editorial peer review in US medical journals,” Journal of the American Medical Association, 263, 1344-1347. Wells, W. D. (1993), “Discovery-oriented consumer research,” Journal of Consumer Research, 19, 498-504. Wensley, R. (1994), “Making better decisions: The challenge of marketing strategy techniques,” International Journal of Research in Marketing, 11, 85-90. Wolin, L. (1962), "Responsibility for raw data," American Psychologist, 17, 657-658 Yankauer, A. (1990), "Who are the peer reviewers and how much do they review?" Journal of the American Medical Association, 263, 1338-1340. citation: Armstrong, J. Scott (2002) Discovery and Communication of Important Marketing Findings: Evidence and Proposals. [Journal (Paginated)] document_url: http://cogprints.org/5181/1/Discovery_and_Communication.pdf