Abstract:

Aiming at the problems of excessive redundancy and irrelevant features, low training feature information and low parallelization efficiency in big data random forest algorithm based on MapReduce, this paper proposes a parallel random forest algorithm based on information theory and norm (PRFITN). Firstly, the algorithm designs the DRIGFN (dimension reduction based on information gain and Frobenius norm) strategy to reduce the number of redundant and irrelevant features. Secondly, a feature grouping strategy based on information theory (FGSIT) is proposed. According to the FGSIT strategy, the features are grouped, and the stratified sampling method is adopted to ensure the information amount of the training features when constructing the decision tree in the random forest. Accuracy of classification results is improved. Finally, in order to improve the parallel efficiency of the cluster, the redistribution of key-value pairs (RSKP) is presented to realize the rapid and uniform distribution of key-value pairs, and obtain the global classification results. Experimental results show that the algorithm has better classification effect in big data environment, especially for datasets with more features.

Key words: MapReduce, random forest (RF), DRIGFN strategy, feature grouping strategy based on information theory (FGSIT), redistribution of key-value pairs (RSKP) strategy

摘要：

针对MapReduce框架下的随机森林算法在处理大数据问题时存在的冗余与不相关特征过多,训练特征信息量低以及并行化效率低等问题,提出了大数据下基于信息论和范数的并行随机森林算法（PRFITN）。首先,该算法基于信息增益和Frobenius范数设计了一种混合降维策略（DRIGFN）,获得降维后的数据集,有效减少了冗余及不相关特征数;其次,提出了基于信息论的特征分组策略（FGSIT）,根据FGSIT策略将特征分组,采用分层抽样方法,保证了随机森林中决策树构建时训练特征的信息量,提高了分类结果的准确度;最后,在Reduce阶段提出了一种键值对重分配策略（RSKP）,获取全局的分类结果,实现了键值对的快速均匀分配,从而提高了集群的并行效率。实验结果表明,该算法在大数据环境下,尤其是针对特征数较多的数据集有更好的分类效果。

关键词: MapReduce框架, 随机森林（RF）, DRIGFN策略, 基于信息论的特征分组策略（FGSIT）, 键值对重分配策略（RSKP）