On entropy-based term weighting schemes for text categorization

Tao Wang, Yi Cai*, Ho-fung LEUNG, Raymond Y. K LAU, Haoran Xie, Qing Li

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

7 Citations (Scopus)

Abstract

In text categorization, Vector Space Model (VSM) has been widely used for representing documents, in which a document is represented by a vector of terms. Since different terms contribute to a document’s semantics in various degrees, a number of term weighting schemes have been proposed for VSM to improve text categorization performance. Much evidence shows that the performance of a term weighting scheme often varies across different text categorization tasks, while the mechanism underlying variability in a scheme’s performance remains unclear. Moreover, existing schemes often weight a term with respect to a category locally, without considering the global distribution of a term’s occurrences across all categories in a corpus. In this paper, we first systematically examine pros and cons of existing term weighting schemes in text categorization and explore the reasons why some schemes with sound theoretical bases, such as chi-square test and information gain, perform poorly in empirical evaluations. By measuring the concentration that a term distributes across all categories in a corpus, we then propose a series of entropy-based term weighting schemes to measure the distinguishing power of a term in text categorization. Through extensive experiments on five different datasets, the proposed term weighting schemes consistently outperform the state-of-the-art schemes. Moreover, our findings shed new light on how to choose and develop an effective term weighting scheme for a specific text categorization task.
Original languageEnglish
Pages (from-to)2313-2346
Number of pages34
JournalKNOWLEDGE AND INFORMATION SYSTEMS
Volume63
Issue number9
DOIs
Publication statusPublished - Sept 2021

Fingerprint

Dive into the research topics of 'On entropy-based term weighting schemes for text categorization'. Together they form a unique fingerprint.

Cite this