Improved Two-View Random Forest

doi:10.3778/j.issn.1673-9418.2008038

Abstract

Abstract:

Random forest (RF) is one of the most classic machine learning methods, which has been widely used. However, although there are many two-view data in reality and extensive analytical research has been carried out, the RF construction for two-view scenarios is little. The only RF method for two-view learning first generates RF for each view respectively, and then merges the view information when making decisions. Therefore, it turns out an obvious disadvantage that the correlation between views is not utilized effectively during the RF construction stage, which undoubtedly wastes information resources. In order to make up for this disadvantage, an improved two-view RF (ITVRF) is proposed in this paper. Specifically, canonical correlation analysis (CCA) is used for view fusion in the process of generating decision trees, and the information interaction between views is embedded into the tree construction stage, realizing the utilization of complementary information between views in the entire RF generation process. In addition, ITVRF also generates discriminant decision boundaries for decision trees through discriminant analysis and thus makes it more suitable for classification. Experimental results show that ITVRF achieves better accuracy than existing two-view RF (TVRF).

Key words: decision tree, random forest (RF), two-view learning, canonical correlation analysis (CCA)

摘要：

随机森林（RF）是最经典的机器学习算法之一,并已获得广泛应用。然而观察发现,尽管现实中存在众多的二视图数据并已获得广泛的分析研究,但针对二视图场景的RF构建相当少,仅有的利用RF解决二视图学习问题的方法也都是先为各个视图生成各自的RF,在决策时才融合了视图间的信息。这样的方法存在一个显著不足是在其RF的构建阶段未利用两个视图间的相关性,这无疑浪费了信息资源。为了弥补这一不足,提出了一种改进的二视图随机森林（ITVRF）。具体而言,在决策树的生成过程中采用典型相关分析（CCA）进行视图融合,将视图间的信息交互融入到了决策树的构建阶段,实现了视图间互补信息在整个RF生成过程中的利用。此外,ITVRF还通过判别分析为决策树生成判别决策边界,更适合于分类。实验结果表明ITVRF比现有的二视图RF（TVRF）有着更优的准确率。

关键词: 决策树, 随机森林（RF）, 二视图学习, 典型相关分析（CCA）

CLC Number:

TP391

XIA Xiaoqiu, CHEN Songcan. Improved Two-View Random Forest[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(1): 144-152.

夏笑秋, 陈松灿. 改进的二视图随机森林[J]. 计算机科学与探索, 2022, 16(1): 144-152.

Figures/Tables 9

Table 1 Statistics for UCI datasets

Dataset	Size	Feature	Class 1	Class 2
Iris	100	4	50	50
Banknote	1 372	4	762	610
Ionosphere	351	34	126	225
WBC	683	9	444	239
Seeds	140	7	70	70
Pima	768	8	268	500
Blood	748	4	570	178
Diabetes	768	4	500	258
WDBC	569	30	212	357
Waveform	3 308	40	1 653	1 655
CMC	962	9	629	333
Mushroom	8 124	22	3 916	4 208

Table 2 Robot execution failures dataset

Learning problem	Instances	Classes and distribution
LP1	88	24% normal 19% collision 18% front collision 39% obstruction
LP2	47	43% normal 13% front collision 15% back collision 11% collision to the right 19% collision to the left
LP3	47	43% ok 19% slightly 32% moved 6% lost
LP4	117	21% normal 62% collision 18% obstruction
LP5	164	27% normal 16% bottom collision 13% bottom obstruction 29% collision in part 16% collision in tool

Table 3 Two-view dataset information selected from MSRC-v1 datasets

Dataset	Instances	View	Feature
MSRC1-v1	210	2	24 color moment,576 histogram of oriented gradient
MSRC2-v1	210	2	256 local binary pattern,254 centrist features

Table 4 Experimental simulation parameters

$K$	$depth$	$min_obj$	criterion
10	max	2	information gain

Table 4 Experimental simulation parameters

$K$	$depth$	$min_obj$	criterion
10	max	2	information gain

Table 5 AUC value and running time

Dataset	TVRF		TV_fisherRF		ITVRF
Dataset	AUC	Time/ms	AUC	Time/ms	AUC	Time/ms
Iris	80.2%±0.08	1	84.5%±0.08	1	92.7%±0.05	1
Banknote	75.1%±0.02	400	75.2%±0.08	600	94.7%±0.01	600
Ionosphere	84.8%±0.03	500	82.2%±0.08	350	88.6%±0.03	450
WBC	86.2%±0.01	430	87.3%±0.01	590	90.2%±0.03	550
Seeds	92.3%±0.01	1	92.5%±0.01	1	93.0%±0.01	1
Pima	66.6%±0.05	500	67.8%±0.06	500	68.3%±0.05	450
Blood	63.7%±0.31	450	65.6%±0.38	800	66.2%±0.3	800
Diabetes	67.0%±0.04	200	68.3%±0.07	900	69.3%±0.06	800
WDBC	87.3%±0.02	500	90.9%±0.01	400	90.5%±0.02	400
Waveform	80.8%±0.03	3 800	81.0%±0.03	650	82.5%±0.03	550
CMC	49.8%±0.19	450	48.6%±0.10	900	50.3%±0.09	850
Mushroom	97.2%±0.01	900	97.9%±0.02	900	100.0%±0.00	800
SPECTF	51.6%±0.01	480	58.8%±0.01	600	71.4%±0.02	650
机器人执行故障	91.9%±0.01	500	92.1%±0.04	350	97.3%±0.01	350
MSRC1-v1	91.4%±0.02	7 200	92.2%±0.04	5 500	93.5%±0.01	4 500
MSRC2-v1	90.6%±0.04	7 800	91.1%±0.05	6 200	94.2%±0.02	6 600

Table 6 AUC values of ITVRF and multi-view method MLRA

Dataset	MLRA	ITVRF
Iris	84.2%±0.02	92.7%±0.05
Ionosphere	86.7%±0.03	88.6%±0.03
Pima	73.9%±0.03	68.3%±0.05
WDBC	93.1%±0.01	90.5%±0.02
Waveform	76.8%±0.02	82.5%±0.03

Fig.1 AUC value with different values of K

Fig.2 AUC value with different values of depth

Fig.3 AUC value with different values of min_obj

References 27

[1]	BREIMAN L. Random forests[J]. Machine Learning, 2001, 45(1):5-32. DOI URL
[2]	BIAU G, SCORNET E. A random forest guided tour[J]. Test, 2016, 25(2):197-227. DOI URL
[3]	BOULESTEIX A L, JANITZA S, KRUPPA J, et al. Over-view of random forest methodology and practical guidance with emphasis on computational biology and bioinformatics[J]. Wiley Interdisciplinary Reviews: Data Mining and Know-ledge Discovery, 2012, 2(6):493-507.
[4]	KONTSCHIEDER P, FITERAU M, CRIMINISI A, et al. Deep neural decision forests[C]// Proceedings of the 2015 IEEE International Conference on Computer Vision, Santiago, Dec 13-16, 2015. Washington: IEEE Computer Society, 2015: 1467-1475.
[5]	COOTES T F, IONITA M C, LINDNER C, et al. Robust and accurate shape model fitting using random forest reg-ression voting[C]// LNCS 7578: Proceedings of the 12th European Conference on Computer Vision, Firenze, Oct 7-13, 2012. Berlin, Heidelberg: Springer, 2012: 278-291.
[6]	BIFET A, HOLMES G, PFAHRINGER B, et al. New ense-mble methods for evolving data streams[C]// Proceedings of the 15th ACM SIGKDD International Conference on Know-ledge Discovery and Data Mining, Paris, Jun 28-Jul 1, 2009. New York: ACM, 2009: 139-148.
[7]	XIONG C M, JOHNSON D M, XU R, et al. Random forests for metric learning with implicit pair-wise position dependence[C]// Proceedings of the 18th ACM SIGKDD Inter-national Conference on Knowledge Discovery and Data Mining, Beijing, Aug 12-16, 2012. New York: ACM, 2012: 958-966.
[8]	DENIL M, MATHESON D, DE FREITAS N. Narrowing the gap: random forests in theory and in practice[C]// Procee-dings of the 31st International Conference on Machine Learning, Beijing, Jun 21-26, 2014: 665-673.
[9]	WANG Y S, XIA S T, TANG Q T, et al. A novel consistent random forest framework: Bernoulli random forests[J]. IEEE Transactions on Neural Networks and Learning Sys-tems, 2018, 29(8):3510-3523.
[10]	BERNARD S, ADAM S, HEUTTE L. Dynamic random forests[J]. Pattern Recognition Letters, 2012, 33(12):1580-1586. DOI URL
[11]	ZHOU Z H, FENG J. Deep forest: towards an alternative to deep neural networks[C]// Proceedings of the 26th Inter-national Joint Conference on Artificial Intelligence, Mel-bourne, Aug 19-25, 2017. Menlo Park: AAAI, 2017: 3553-3559.
[12]	SUN S L. A survey of multi-view machine learning[J]. Neu-ral Computing and Applications, 2013, 23(7/8):2031-2038.
[13]	XU C, TAO D C, XU C. A survey on multi-view learning[J]. arXiv:1304.5634, 2013.
[14]	BLUM A, MITCHELL T M. Combining labeled and un-labeled data with cotraining[C]// Proceedings of the 11th Annual Conference on Computational Learning Theory, Madison, Jul 24-26, 1998. New York: ACM, 1998: 92-100.
[15]	MUSLEA I, MINTON S, KNOBLOCK C A. Active lear-ning with multiple views[J]. Journal of Artificial Intelli-gence Research, 2006, 27(1):203-233.
[16]	SUN S L, SHAWE-TAYLOR J. Sparse semi-supervised learning using conjugate functions[J]. Journal of Machine Learning Research, 2010, 11:2423-2455.
[17]	SUN S L, CHAO G Q. Multi-view maximum entropy discrimination[C]// Proceedings of the 23rd International Joint Conference on Artificial Intelligence, Beijing, Aug 3-9, 2013. Menlo Park: AAAI, 2013: 1706-1712.
[18]	GONZÁLEZ A, VILLALONGA G, XU J L, et al. Multi-view random forest of local experts combining RGB and LIDAR data for pedestrian detection[C]// Proceedings of the 2018 IEEE Intelligent Vehicles Symposium, Seoul, Jun 28-Jul 1, 2015. Piscataway: IEEE, 2015: 356-361.
[19]	CAO H L, BERNARD S, SABOURIN R, et al. Random forest dissimilarity based multiview learning for radiomics application[J]. Pattern Recognition, 2019, 88:185-197. DOI URL
[20]	HARDOON D R, SZEDMÁK S, SHAWE-TAYLOR J. Ca-nonical correlation analysis: an overview with application to learning methods[J]. Neural Computation, 2004, 16(12):2639-2664. DOI URL
[21]	SUN Q S, ZENG S G, LIU Y, et al. A new method of fea-ture fusion and its application in image recognition[J]. Pat-tern Recognition, 2005, 38(12):2437-2448.
[22]	FISHER R A. The use of multiple measurements in tax-onomic problems[J]. Annals of Eugenics, 1936, 7(2):179-188. DOI URL
[23]	周旭东, 陈晓红, 陈松灿. 增强组合特征判别性的典型相关分析[J]. 模式识别与人工智能, 2012, 25(2):285-291.
	ZHOU X D, CHEN X H, CHEN S C. Combined-feature-discriminability enhanced canonical correlation analysis[J]. Pattern Recognition and Artificial Intelligence, 2012, 25(2):285-291.
[24]	SUN T K, CHEN S C, YANG J Y, et al. A novel method of combined feature extraction for recognition[C]// Proceedings of the 8th IEEE International Conference on Data Mining, Pisa, Dec 15-19, 2008. Washington: IEEE Computer Society, 2008: 1043-1048.
[25]	LI S, SHAO M, FU Y. Multi-view low-rank analysis for outlier detection[C]// Proceedings of the 2015 SIAM Interna-tional Conference on Data Mining, Vancouver, Apr 30-May 2, 2015. Philadelphia: SIAM, 2015: 748-756.
[26]	ZHAO H D, FU Y. Dual-regularized multi-view outlier de-tection[C]// Proceedings of the 24th International Joint Conference on Artificial Intelligence, Buenos Aires, Jul 25-31, 2015. Menlo Park: AAAI, 2015: 4077-4083.
[27]	LÓPEZ-CHAU A, CERVANTES J, LÓPEZ-GARCÍA L, et al. Fisher’s decision tree[J]. Expert Systems with Applica-tions, 2013, 40(16):6283-6291.