creators_name: Turney, Peter D. creators_name: Littman, Michael L. creators_name: Bigham, Jeffrey creators_name: Shnayder, Victor creators_id: 2175 creators_id: creators_id: creators_id: editors_name: Nicolov, Nicolas editors_name: Botcheva, Kalina editors_name: Angelova, Galia editors_name: Mitkov, Ruslan type: bookchapter datestamp: 2005-01-10 lastmod: 2011-03-11 08:55:49 metadata_visibility: show title: Combining Independent Modules in Lexical Multiple-Choice Problems ispublished: pub subjects: comp-sci-stat-model subjects: comp-sci-lang subjects: ling-comput subjects: ling-sem subjects: comp-sci-mach-learn full_text_status: public abstract: Existing statistical approaches to natural language problems are very coarse approximations to the true complexity of language processing. As such, no single technique will be best for all problem instances. Many researchers are examining ensemble methods that combine the output of multiple modules to create more accurate solutions. This paper examines three merging rules for combining probability distributions: the familiar mixture rule, the logarithmic rule, and a novel product rule. These rules were applied with state-of-the-art results to two problems used to assess human mastery of lexical semantics -- synonym questions and analogy questions. All three merging rules result in ensembles that are more accurate than any of their component modules. The differences among the three rules are not statistically significant, but it is suggestive that the popular mixture rule is not the best rule for either of the two problems. date: 2004 date_type: published publication: Recent Advances in Natural Language Processing III: Selected Papers from RANLP 2003 publisher: John Benjamins pagerange: 101-110 refereed: TRUE referencetext: Brill, Eric & Jun Wu. 1998. "Classifier Combination for Improved Lexical Disambiguation". Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL’98), vol. 1, 191-195. Montreal, Canada. Chalmers, David J., Robert M. French, & Douglas R. Hofstadter. 1992. "High-level Perception, Representation and Analogy: A Critique of Artificial Intelligence Methodology". Journal of Experimental and Theoretical Artificial Intelligence 4:185-211. Fellbaum, Christiane. 1998. WordNet: An Electronic Lexical Database. Cambridge, Mass.: MIT Press. Florian, Radu & David Yarowsky. 2002. "Modeling Consensus: Classifier Combination for Word Sense Disambiguation". Proceedings of the Conference on Empirical Methods in Natural Language Processing, 25-32. Philadelphia. Heskes, Tom. 1998. "Selecting Weighting Factors in Logarithmic Opinion Pools". Advances in Neural Information Processing Systems 10:266-272. Hinton, Geoffrey E. 1999. "Products of Experts". Proceedings of the 9th International Conference on Artificial Neural Networks (ICANN 99), 1:1-6. Edinburgh, Scotland. Jacobs, Robert A. 1995. "Methods for Combining Experts’ Probability Assessments". Neural Computation 7:5.867-888. Jacobs, Robert A., Michael I. Jordan, Steve J. Nowlan & Geoffrey E. Hinton. 1991. "Adaptive Mixtures of Experts". Neural Computation 3:79-87. Jarmasz, Mario & Stan Szpakowicz. 2003. "Roget’s Thesaurus and Semantic Similarity". Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP-03), 212-219. Borovets, Bulgaria. Lakoff, George & Mark Johnson. 1980. Metaphors We Live By. Chicago: University of Chicago Press. Landauer, Thomas K. & Susan T. Dumais. 1997. "A Solution to Plato’s Problem: The Latent Semantic Analysis Theory of Acquisition, Induction and Representation of Knowledge". Psychological Review 104:2.211-240. Littman, Michael L., Greg A. Keim & Noam Shazeer. 2002. "A Probabilistic Approach to Solving Crossword Puzzles". Artificial Intelligence 134:23-55. Schapire, Robert E. 1999. "A Brief Introduction to Boosting". Proceedings of the 16th International Joint Conference on Artificial Intelligence (IJCAI-99), 1401-1406. Stockholm, Sweden. Terra, Egidio & C. L.A. Clarke. 2003. "Frequency Estimates for Statistical Word Similarity Measures". Proceedings of the Human Language Technology and North American Chapter of Association of Computational Linguistics Conference 2003 (HLT/NAACL 2003), 244-251. Edmonton, Alberta, Canada. Turney, Peter D. 2001. "Mining the Web for Synonyms: PMI-IR versus LSA on TOEFL". Proceedings of the 12th European Conference on Machine Learning (ECML-2001), 491-502. Freiburg, Germany. Turney, Peter D. & Michael L. Littman. 2003a. "Learning Analogies and Semantic Relations". Technical Report ERB-1103. Ottawa, Ontario, Canada: National Research Council, Institute for Information Technology. Turney, Peter D. & Michael L. Littman. 2003b. "Measuring Praise and Criticism: Inference of Semantic Orientation from Association". ACMTransactions on Information Systems 21:4.315-346. Turney, Peter D., Michael L. Littman, Jeffrey Bigham & Victor Shnayder. 2003. "Combining Independent Modules to SolveMultiple-Choice Synonym and Analogy Problems". Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP-03), 482-489. Borovets, Bulgaria. Xu, Lei, Adam Krzyzak & Ching Y. Suen. 1992. "Methods of Combining Multiple Classifiers and Their Applications to Handwriting Recognition". IEEE Transactions on Systems, Man and Cybernetics 22:3.418-435. citation: Turney, Peter D. and Littman, Michael L. and Bigham, Jeffrey and Shnayder, Victor (2004) Combining Independent Modules in Lexical Multiple-Choice Problems. [Book Chapter] document_url: http://cogprints.org/4027/1/turney.pdf