Least Squares Support Vector Machine for Minimizing VC Dimensional Expectation Upper Bound

doi:10.3778/j.issn.1673-9418.2112108

Abstract

Abstract: Machine learning is an important aspect of modern computer technology, and the support vector machine method has been widely used in all walks of life because of its good performance in recent years. The performance of support vector machine can be measured by VC (Vapnik-Chervonenkis) dimension, which is an index to measure the complexity of the machine. In theory, low VC dimension can be well generalized. However, for some classifier methods based on traditional support vector machine, the upper bound of VC dimension of support vector machine may be infinite when dealing with a variety of data. Although good results have been obtained in practice and application, it can not guarantee good generalization, resulting in poor results for some special data. Therefore, this paper proposes an improved LSSVM (least squares support vector machines) algorithm. Based on LSSVM algorithm, this paper minimizes the upper bound of VC dimension and finds the desired best projection scheme. Finally, this paper brings it into LSSVM algorithm to classify the data. Experimental results show that the error rate of the classifier is lower than that of the traditional least squares support vector machine on the benchmark dataset, which means that the proposed algorithm makes the test accuracy better than the comparison algorithm with the approximate number of support vectors, and improves the generalization ability of the algorithm.

Key words: least squares support vector machine (LSSVM), machine learning, VC dimension, classification

摘要： 机器学习是现代计算机技术中比较重要的一个方面，而其中的支持向量机方法也因其良好的性能近年来得到了广泛关注，被应用于各行各业当中。而支持向量机的性能可以用VC维来衡量，VC维是衡量机器复杂性的一种指标，理论上来说，低VC维可以得到很好的泛化。但是，对于以传统支持向量机为基础的一些分类器方法来说，VC维在处理各种各样的数据时，支持向量机的VC维的上界可能是无穷的。尽管在实践与应用中得到了很好的结果，但并不能保证很好的泛化，导致对于一些特殊的数据取得的效果不好。因此，提出了一种改进LSSVM的算法，以LSSVM算法为基础，将VC维上界最小化并找到其期望的最佳投影方案，最后带入到LSSVM算法中来对数据进行分类。实验结果表明，在所采用的基准数据集上，该分类器的错误率低于传统最小二乘支持向量机，这意味着，所提出的算法以近似的支持向量个数使得测试精度优于比较算法，提高了算法的泛化能力。

关键词: 最小二乘支持向量机（LSSVM）, 机器学习, VC维, 分类

LI Hao, WANG Shitong. Least Squares Support Vector Machine for Minimizing VC Dimensional Expectation Upper Bound[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(7): 1599-1608.

李昊, 王士同. VC维期望上界最小化的最小二乘支持向量机[J]. 计算机科学与探索, 2023, 17(7): 1599-1608.

References

[1] CORTES C, VAPNIK V. Support-vector networks[J]. Machine Learning, 1995, 20 (3): 273-297.
[2] YANG L, YANG S, ZHANG R, et al. Sparse least square support vector machine via coupled compressive pruning[J]. Neurocomputing, 2014, 131: 77-86.
[3] DING S, HUA X. Recursive least squares projection twin support vector machines for nonlinear classification[J].Neurocomputing, 2014, 130: 3-9.
[4] WU D, SHAO L. Multi-max-margin support vector machine for multi-source human action recognition[J]. Neurocomputing, 2014, 127: 98-103.
[5] 林浩, 李雷孝, 王慧. 支持向量机在智能交通系统中的研究应用综述[J]. 计算机科学与探索, 2020, 14(6): 901-917.
LIN H, LI L X, WANG H. Survey on research and application of support vector machines in intelligent transportation system[J]. Journal of Frontiers of Computer Science and Technology, 2020, 14(6): 901-917.
[6] 程凤伟, 王文剑. 基于近邻传输的粒度SVM算法[J]. 计算机科学与探索, 2020, 14(7): 1194-1199.
CHENG F W, WANG W J. Granular support vector machine algorithm based on affinity propagation[J]. Journal of Frontiers of Computer Science and Technology, 2020, 14(7): 1194-1199.
[7] 何丽, 韩克平, 刘颖. 自适应的SVM增量算法[J]. 计算机科学与探索, 2019, 13(4): 647-656.
HE L, HAN K P, LIU Y. Self-adaptive SVM incremental learning algorithm[J]. Journal of Frontiers of Computer Science and Technology, 2019, 13(4): 647-656.
[8] SUYKENS J, VANDEWALLE J. Least squares support vector machine classifiers[J]. Neural Processing Letters, 1999, 9(3): 293-300.
[9] ZHANG Y, YAN S, QIAN X, et al. A fault diagnosis based on LSSVM and Bayesian probability for wind turbines[C]//Proceedings of the 2020 39th Chinese Control Conference,Shenyang, Jul 27-29, 2020. Piscataway: IEEE, 2020: 4101-4106.
[10] XIU L, NING W. RNA-GA with whale search based LSSVMs for modeling multivariable systems[C]//Proceedings of the 37th Chinese Control Conference, Wuhan, Jul 25-27, 2018.Piscataway: IEEE, 2018: 739-744.
[11] RAZAK I, ABIDIN I Z, SIAH Y K, et al. An optimization method of genetic algorithm for LSSVM in medium term electricity price forecasting[J]. Journal of Telecommunication, Electronic and Computer Engineering, 2018, 10: 99-103.
[12] YU C, XI Z, LU Y, et al. LSSVM-based color prediction for cotton fabrics with reactive pad-dry-pad-steam dyeing[J]. Chemometrics and Intelligent Laboratory Systems, 2020, 199(3): 103956.
[13] 杨建新, 兰小平, 姚志强, 等. 基于郊狼算法优化的LSSVM多工序质量预测方法[J]. 制造业自动化, 2021, 43(12):105-109.
YANG J X, LAN X P, YAO Z Q, et al. Multi-process product quality prediction based on LSSVM optimized by coyote optimization algorithm[J]. Manufacturing Automation, 2021, 43(12): 105-109.
[14] BURGES C. A tutorial on support vector machines for pattern recognition[J]. Data Mining and Knowledge Discovery, 1998, 2(2): 121-167.
[15] JAYADEVA. Learning a hyperplane classifier by minimizing an exact bound on the VC dimension[J]. Neurocomputing, 2014, 149: 683-689.
[16] 林建忠. 金融信息分析[M]. 上海: 上海交通大学出版社, 2015.
LIN J Z. Financial information analysis[M]. Shanghai: Shanghai Jiaotong University Press, 2015.
[17] 刘春. 基于PSO-LSSVM的网络流量预测模型[J]. 计算机系统应用, 2014, 23(10): 147-151.
LIU C. Network traffic prediction method based on particle swarm algorithm optimizing least square support vector machine[J]. Computer Systems & Applications, 2014, 23(10): 147-151.