Incremental Reduced Least Squares Twin Support Vector Regression

doi:10.3778/j.issn.1673-9418.1912005

Abstract

Abstract:

In the incremental least squares twin support vector regression, to solve the problem that the constituted kernel matrix cannot approximate the original kernel matrix well, this paper proposes an incremental reduced least squares twin support vector regression (IRLSTSVR) algorithm. Firstly, in order to reduce the correlation of column vectors in the kernel matrix, the proposed algorithm utilizes a reduced method to determine the correlation between column vectors and then screen support vectors from samples to constitute the column vectors of the kernel matrix. Therefore, the constituted kernel matrix can better approximate the original counterpart, which ensures the sparsity of the solution. Secondly, the inverse matrix is incrementally updated by the block matrix inverse lemma, which further shortens the training time of the proposed algorithm. Finally, the feasibility and efficacy of the proposed algorithm are verified on the benchmark datasets. Experimental results show that the IRLSTSVR algorithm can obtain sparse solution and its generalization performance is closer to offline algorithm compared with state-of-the-art algorithms.

Key words: least squares, twin support vector regression (TSVR), reduced method, incremental learning

摘要：

为了解决增量式最小二乘孪生支持向量回归机存在构成的核矩阵无法很好地逼近原核矩阵的问题，提出了一种增量式约简最小二乘孪生支持向量回归机（IRLSTSVR）算法。该算法首先利用约简方法，判定核矩阵列向量之间的相关性，筛选出用于构成核矩阵列向量的样本作为支持向量以降低核矩阵中列向量的相关性，使得构成的核矩阵能够更好地逼近原核矩阵，保证解的稀疏性。然后通过分块矩阵求逆引理高效增量更新逆矩阵，进一步缩短了算法的训练时间。最后在基准测试数据集上验证算法的可行性和有效性。实验结果表明，与现有的代表性算法相比，IRLSTSVR算法能够获得稀疏解和更接近离线算法的泛化性能。

关键词: 最小二乘, 孪生支持向量回归机（TSVR）, 约简方法, 增量式学习

CAO Jie, GU Binjie, XIONG Weili, PAN Feng. Incremental Reduced Least Squares Twin Support Vector Regression[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(3): 553-563.

曹杰, 顾斌杰, 熊伟丽, 潘丰. 增量式约简最小二乘孪生支持向量回归机[J]. 计算机科学与探索, 2021, 15(3): 553-563.

References

[1] VAPNIK V N. The nature of statistical learning theory[M]. Berlin, Heidelberg: Springer, 1995.
[2] WANG L T, ZHANG J X, ZHANG J H. Application of crow search algorithm in SVM parameter optimization[J]. Com-puter Engineering and Applications, 2019, 55(21): 214-219.
王丽婷, 张金鑫, 张金华. 乌鸦搜索算法在SVM参数优化中的应用[J]. 计算机工程与应用, 2019, 55(21): 214-219.
[3] LIU Z W, LIU H Q, ZHAO Z K. Weighted least squares support vector machine for semi-supervised classification[J]. Wireless Personal Communications, 2018, 103(3): 1-12.
[4] BURGES C J C. A tutorial on support vector machines for pattern recognition[J]. Data Mining and Knowledge Discovery, 1998, 2(2): 121-167.
[5] LILLEBERG J, YUN Z, ZHANG Y. Support vector mach-ines and word2vec for text classification with semantic features[C]//Proceedings of the 14th IEEE International Conference on Cognitive Informatics & Cognitive Computing, Beijing, Jul 6-8, 2015. Washington: IEEE Computer Society, 2015: 136-140.
[6] ZHENG B, MYINT S W, THENKABAIL P S, et al. A support vector machine to identify irrigated crop types using time-series landsat NDVI data[J]. International Journal of Applied Earth Observations & Geoinformation, 2015, 34(1): 103-112.
[7] HU Z H, XU Y W, ZHAO X L, et al. Multi-feature selection tracking based on support vector machine[J]. Journal of Applied Sciences, 2015, 33(5): 502-517.
胡昭华, 徐玉伟, 赵孝磊, 等. 基于支持向量机的多特征选择目标跟踪[J]. 应用科学学报, 2015, 33(5): 502-517.
[8] GAN L, YANG M. Pedestrian detection method based on ensemble SVM classifier[J]. Computer Engineering and Applications, 2019, 55(7): 194-198.
甘玲, 杨梦. 聚合支持向量机分类器的行人检测方法[J]. 计算机工程与应用, 2019, 55(7): 194-198.
[9] COLLOBERT R, BENGIO S. SVMTorch: support vector machines for large-scale regression problems[J]. Journal of Machine Learning Research, 2001, 1: 143-160.
[10] JAYADEVA, KHEMCHANDANI R, CHANDRA S. Twin support vector machines for pattern classification[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(5): 905-910.
[11] PENG X. TSVR: an efficient twin support vector machine for regression[J]. Neural Networks, 2010, 23(3): 365-372.
[12] SINGH M, CHADHA J, AHUJA P, et al. Reduced twin support vector regression[J]. Neurocomputing, 2010, 74(9): 1474-1477.
[13] SHAO Y H, ZHANG C H, YANG Z M, et al. An ε-twin support vector machine for regression[J]. Neural Computing & Applications, 2013, 23(1): 175-185.
[14] LU Z X, YANG Z X, GAO X Y. Least square twin support vector regression[J]. Computer Engineering and Applications, 2014, 50(23): 140-144.
卢振兴, 杨志霞, 高新豫. 最小二乘双支持向量回归机[J]. 计算机工程与应用, 2014, 50(23): 140-144.
[15] RASTOGI R, ANAND P, CHANDRA S. A ν-twin support vector machine based regression with automatic accuracy control[J]. Applied Intelligence, 2016, 46(3): 1-14.
[16] CAUWENBERGHS G, POGGIO T A. Incremental and decremental support vector machine learning[C]//Proceedings of the 2000 International Conference on Neural Information Processing Systems, Denver, Jan 1-3, 2000. Cambridge: MIT Press, 2000: 388-394.
[17] HE L, HAN K P, LIU Y. Self-adaptive SVM incremental learning algorithm[J]. Journal of Frontiers of Computer Science and Technology, 2019, 13(4): 647-656.
何丽, 韩克平, 刘颖. 自适应的SVM增量算法[J]. 计算机科学与探索, 2019, 13(4): 647-656.
[18] MA J S, JAMES T, SIMON P. Accurate online support vector regression[J]. Neural Computation, 2003, 15(11): 2683- 2703.
[19] GU B J, PAN F. Accurate incremental online ν-support vector regression learning algorithm[J]. Control Theory & Applications, 2016, 33(4): 466-478.
顾斌杰, 潘丰. 精确增量式在线ν型支持向量回归机学习算法[J]. 控制理论与应用, 2016, 33(4): 466-478.
[20] ZHANG H R, WANG X D. Incremental and online learning algorithm for regression least squares support vector machine[J]. Chinese Journal of Computers, 2006, 29(3): 400-406.
张浩然, 汪晓东. 回归最小二乘支持向量机的增量和在线式学习算法[J]. 计算机学报, 2006, 29(3): 400-406.
[21] ZHAO Y P, DU Z H, ZHANG Z A, et al. Online independent reduced least squares support vector regression[J]. Information Sciences, 2012, 201(19): 37-52.
[22] HAO Y H, ZHANG H F. Incremental learning algorithm based on twin support vector regression[J]. Computer Science, 2016(2): 230-234.
郝运河, 张浩峰. 基于双支持向量回归机的增量学习算法[J]. 计算机科学, 2016(2): 230-234.
[23] TYLAVSKY D J, SOHIE G R L. Generalization of the matrix inversion lemma[J]. Proceedings of the IEEE, 1986, 74(7): 1050-1052.
[24] GOLUB G H, VAN LOAN C F. Matrix computations[M]. Baltimore: Johns Hopkins University Press, 1996.
[25] SUYKENS J A K, VANDEWALE J. Least squares support vector machine classifiers[J]. Neural Processing Letters, 1999, 9(3): 293-300.
[26] BALASUNDARAM S, TANVEER M. On Lagrangian twin support vector regression[J]. Neural Computing and Appli-cations, 2013, 22(S1): 257-267.
[27] TANVEER M, SHUBHAM K, AL-DHAIFALLAH M, et al. An efficient regularized K-nearest neighbor-based weighted twin support vector regression[J]. Knowledge-Based Systems, 2016, 94: 70-87.
[28] TANVEER M, SHUBHAM K. A regularization on Lagrangian twin support vector regression[J]. International Journal of Machine Learning and Cybernetics, 2017, 8(3): 807-821.