关键词: 特征词权重, 熵加权, 文本分类, 类别区分力

Abstract: As the volume of textual data has become very large and is still increasing rapidly, automatic text categorization (TC) is becoming more and more important. Term weighting or feature weight calculation is one of the hot research topics in TC to improve the classification accuracy. It is found that entropy-based weighting (EW) methods are usually more effective than others. However, there are still some problems with the existing EW methods, e.g., they may perform worse than the traditional TF-IDF (term frequency & inverse document frequency), for TC on some text corpora. So this paper proposes a new term weighting scheme called LTF-ECDP, which combines logarithmic term frequency and entropy-based class distinguishing power as a new weighting factor. In order to test LTP-ECDP and compare it with other weighting methods, a considerable number of TC experiments using support vector machine (SVM) have been done on three popular benchmark datasets including a Chinese corpus, TanCorp, and two English corpora such as WebKB and 20 Newsgroups. The experimental results show that LTF-ECDP outperforms the other five entropy-based weighting methods and two famous methods such as TF-IDF and TF-RF (term frequency & relevance frequency). Compared with the other term weighting methods, LTF-ECDP can further improve the accuracy of TC while keeping good performance on different datasets consistently.

Key words: term weighting, entropy-based weighting, text categorization, class distinguishing power