Improved Fuzzy Rank Mutual Information for Monotonic Feature Selection

doi:10.3778/j.issn.1673-9418.2104114

Abstract

Abstract: The large amount of low-quality information, such as missing values and outliers, makes monotonic classification tasks usually not satisfy the monotonic consistency. However, for the existing ordered feature selection algorithm, the mutual information used to evaluate the feature relevance doesn’t satisfy the monotonic restriction between feature sets and the metrics when the monotonic classification tasks are inconsistent. In response to the problem, this paper firstly improves the classical fuzzy rank information entropy to make it satisfy the monotonicity restriction. The novel fuzzy rank mutual information (nFRMI) is further defined to evaluate the monotonic consistency of features, and it is proven that the metric criterion satisfies the monotonicity in both consistent and inconsistent monotonic classification tasks. In addition, a two-stage feature selection algorithm is proposed in combination with the minimal-redundancy-maximal-relevance (mRMR) criterion. In the first stage of the algorithm, it calculates significance of each feature based on nFRMI, and then obtains the feature ranking based on the feature significance; the Wrapper method is used to select the feature subset with the optimal classification performance in the second stage. Simulation experiments are conducted on four UCI datasets to verify the effectiveness of the proposed algorithm from the perspective of classification accuracy.

Key words: feature selection, monotonic classification, inconsistent, fuzzy information entropy, rank mutual information

摘要： 缺失值、异常值等低质信息的大量存在使得实际应用中单调分类任务通常不满足一致性单调约束。然而，现有的面向单调分类的特征选择算法中用于评估特征相关性的互信息度量准则在处理不一致单调分类任务时不满足特征集合与度量准则之间的单调性限制关系。针对此问题，首先对经典的模糊排序信息熵进行了改进，使之满足单调性限制关系。进一步定义了模糊排序互信息（nFRMI）用于评估特征之间的单调一致性，并证明了该度量准则在一致性和不一致性单调分类任务中均满足单调性约束，此外结合最大相关最小冗余（mRMR）准则提出了两阶段的特征选择算法。算法第一阶段利用nFRMI计算每个特征的特征重要度，然后根据特征重要度得到特征排序；第二阶段采用Wrapper方法选出具有最优分类性能的特征子集。仿真实验从UCI挑选了四个单调分类数据集，并从分类精度指标对所提出的特征选择算法的有效性进行了验证。

关键词: 特征选择, 单调分类, 不一致, 模糊信息熵, 排序互信息

PI Hong, LUO Chuan, LI Tianrui, CHEN Hongmei. Improved Fuzzy Rank Mutual Information for Monotonic Feature Selection[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(1): 108-115.

皮洪, 罗川, 李天瑞, 陈红梅. 面向单调特征选择的改进模糊排序互信息[J]. 计算机科学与探索, 2023, 17(1): 108-115.

References

[1] HU Q H, PAN W W, ZHANG L, et al. Feature selection for monotonic classification[J]. IEEE Transactions on Fuzzy Systems, 2012, 20(1): 69-81.
[2] QIAN W B, SHU W H. Mutual information criterion for feature selection from incomplete data[J]. Neurocomputing, 2015, 168: 210-220.
[3] DASH M, LIU H. Feature selection for classification[J]. In-telligent Data Analysis, 1997, 1: 131-156.
[4] CHANDRASHKAR G, SAHIN F. A survey on feature selection methods[J]. Computers and Electrical Engineering, 2014, 40: 16-28.
[5] HU Q H, PAN W W, SONG Y P, et al. Large-margin feature selection for monotonic classification[J]. Knowledge-Based Systems, 2012, 31: 8-18.
[6] PAN W W, HU Q H, SONG Y P, et al. Feature selection for monotonic classification via maximizing monotonic dependency[J]. International Journal of Computational Intelligence Systems, 2014, 7(3): 543-555.
[7] PAN W W, HU Q H. An improved feature selection algorithm for ordinal classification[J]. IEICE Transactions on Funda-mentals of Electronics Communications and Computer Sciences, 2016, 99-A(12): 2266-2274.
[8] VILLELA S M, LEITE S D C, XAVIER A E, et al. An ordered search with a large margin classifier for feature selection[J]. Applied Soft Computing Journal, 2021, 100: 106930.
[9] ESTéVEZ P A, TESMER M, PEREZ C A, et al. Normalized mutual information feature selection[J]. IEEE Transactions on Neural Networks, 2009, 20(2): 189-201.
[10] VERGARA J R, ESTéVEZ P A. A review of feature selection methods based on mutual information[J]. Neural Computing and Applications, 2014, 24: 175-186.
[11] KWAK N, CHOI C H. Input feature selection by mutual information based on parzen window[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, 24(12): 1667-1671.
[12] BATTITI R. Using mutual information for selecting features in supervised neural net learning[J]. IEEE Transactions on Neural Networks, 1994, 5(4): 537-550.
[13] PENG H C, LONG F H, DING C. Feature selection based on mutual information: criteria of max-dependency, max-relevance, and min-redundancy[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(8): 1226-1238.
[14] 许行, 梁吉业, 王宝丽. 基于双向有序互信息的单调分类决策树算法[J]. 南京大学学报(自然科学), 2013, 49(5): 628-636.
XU H, LIANG J Y, WANG B L. Bi-direction rank mutual information based decision trees for monotonic classification[J]. Journal of Nanjing University (Natural Science), 2013, 49(5): 628-636.
[15] LIN Y J, HU Q H, LIU J H, et al. Streaming feature selection for multilabel learning based on fuzzy mutual information[J]. IEEE Transactions on Fuzzy Systems, 2017, 25(6): 1491-1507.
[16] DAI J H, WANG W T, XU Q. An uncertainty measure for incomplete decision tables and its applications[J]. IEEE Transactions on Cybernetics, 2013, 43(4): 1277-1289.
[17] ZHANG X, MEI C L, CHEN D G, et al. Feature selection in mixed data: a method using a novel fuzzy rough set-based information entropy[J]. Pattern Recognition, 2016, 56: 1-15.
[18] CANO J R, GUTIERREZ P A, KRAWCZYK B, et al. Monotonic classification: an overview on algorithms, perfor-mance measures and data sets[J]. Neurocomputing, 2019, 341: 168-182.