Extreme Learning Machine for Optimized Affine Transformation Based on Gaussian Distribution

doi:10.3778/j.issn.1673-9418.1912074

Abstract

Abstract:

Extreme learning machine (ELM) is massively mapped to the saturation region of the activation function. Moreover, the input and output of the hidden layer are far from being able to obtain a common distribution method, which gives rise to poor generalization performance. Aiming at this problem, the extreme learning machine that optimizes the affine transformation (AT) in the activation function under the Gaussian distribution is studied. The proposed algorithm introduces a new linear relationship of input data in the hidden layer. The gradient descent algorithm is used to optimize the scaling parameters and translation parameters in the objective function to satisfy the hidden layer output highly obeying the Gaussian distribution. The new method of calculating affine parameters based on the Gaussian distribution can ensure that the hidden nodes are independent of each other while retaining a high degree of dependency. The experimental results show that the output data of the hidden layer do not obey the uniform distribution well in the actual classification dataset and the image regression dataset, but obey the Gaussian distribution trend, which can achieve better experimental results in general. Compared with the original ELM algorithm and the AT-ELM1 algorithm, there are significant improvements in general.

Key words: extreme learning machine (ELM), affine transformation (AT), Gaussian distribution, classification

摘要：

极限学习机（ELM）会大量映射到激活函数的饱和区域，同时隐含层输入与输出远远不能获得共同的分布方式，导致泛化性能大打折扣。针对这一问题，研究了在高斯分布下优化激活函数中仿射变换（AT）的极限学习机，主要思想是在隐含层输入数据上引入新型的线性关系，利用梯度下降算法对误差函数中的缩放参数和平移参数进行优化，以满足隐含层输出能够高度服从高斯分布。基于高斯分布计算仿射参数的方法，能够保证隐节点相互独立的同时，也强调了高度的依赖关系。实验结果表明，在实际分类数据集和图像回归数据集中，隐含层输出数据不能很好地服从均匀分布，但服从高斯分布趋势，总体上能够达到更好的实验效果。与原始ELM算法和AT-ELM1算法比较，均有显著的改善。

关键词: 极限学习机（ELM）, 仿射变换（AT）, 高斯分布, 分类

ZHANG Yi, WANG Shitong. Extreme Learning Machine for Optimized Affine Transformation Based on Gaussian Distribution[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(4): 690-701.

张毅, 王士同. 在高斯分布下优化仿射变换的极限学习机[J]. 计算机科学与探索, 2021, 15(4): 690-701.

References

[1] HUANG G B, ZHU Q Y, SIEW C K. Extreme learning mach-ine: a new learning scheme of feedforward neural networks[J]. Neural Networks, 2004, 2: 985-990.
[2] HUANG Z, YU Y, GU J, et al. An efficient method for traffic sign recognition based on extreme learning machine[J]. IEEE Transactions on Cybernetics, 2016, 47(4): 920-933.
[3] HUANG G B. Extreme learning machine: theory and appli-cations[J]. Neurocomputing, 2006, 70: 489-501.
[4] IOFFE S, SZEGEDY C. Batch normalization: accelerating deep network training by reducing internal covariate shift[J]. arXiv:1502.03167, 2015.
[5] ZUO P Y, WANG S T. Inverse-matrix-free online sequential extreme learning machine[J]. Journal of Frontiers of Computer Science and Technology, 2020, 14(1): 117-124.
左鹏玉, 王士同. 无逆矩阵在线序列极限学习机[J]. 计算机科学与探索, 2020, 14(1): 117-124.
[6] YU H L, QI Y S, YANG X B, et al. Research on class imbal-ance fuzzy weighted extreme learning machine algorithm[J]. Journal of Frontiers of Computer Science and Technology, 2017, 11(4): 619-632.
于化龙, 祁云嵩, 杨习贝, 等. 类不平衡模糊加权极限学习机算法研究[J]. 计算机科学与探索, 2017, 11(4): 619-632.
[7] LIU X, LIN S B, FANG J, et al. Is extreme learning machine feasible? A theoretical assessment (Part I)[J]. IEEE Transactions on Neural Networks and Learning Systems, 2015, 26(1): 7-20.
[8] BISHOP C M. Pattern recognition and machine learning[M]. Berlin, Heidelberg: Springer, 2006.
[9] LUO X, YANG X N, JIANG C W, et al. Timeliness online regularized extreme learning machine[J]. International Journal of Machine Learning and Cybernetics, 2018, 9(3): 465-476.
[10] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. Imagenet classification with deep convolutional neural networks[C]//Proceedings of the 26th Annual Conference on Neural Infor-mation Processing Systems, Lake Tahoe, Dec 3-6, 2012. Red Hook: Curran Associates, 2012: 1106-1114.
[11] HE K M, ZHANG X Y, REN S Q, et al. Delving deep into rectifiers: surpassing human-level performance on imagenet classification[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision, Santiago, Dec 7-13, 2015.Washington: IEEE Computer Society, 2015: 1026-1034.
[12] XU B, WANG N, CHEN T, et al. Empirical evaluation of rectified activations in convolutional network[J]. arXiv:1505.00853, 2015.
[13] ZHANG L, ZHANG D. Evolutionary cost-sensitive extreme learning machine[J]. IEEE Transactions on Neural Networks and Learning Systems, 2016, 28(12): 3045-3060.
[14] HUANG G B, DING X, ZHOU H. Optimization method based extreme learning machine for classification[J]. Neurocom-puting, 2010, 74: 155-163.
[15] LIU X, WANG L, HUANG G B, et al. Multiple kernel extreme learning machine[J]. Neurocomputing, 2015, 149: 253-264.
[16] CAO J W, ZHANG K, YONG H W, et al. Extreme learning machine with affine transformation inputs in an activation function[J]. IEEE Transactions on Neural Networks and Learning Systems, 2019, 30(7): 2093-2107.
[17] HAN F. Improved learning algorithms of SLFN for appro-ximating periodic function[C]//LNCS 5227: Proceedings of the 4th International Conference on Intelligent Computing, Shanghai, Sep 15-18, 2008. Berlin, Heidelberg: Springer,2008: 654-660.
[18] HUANG G B. What are extreme learning machines? Filling the gap between Frank Rosenblatt??s dream and John von Neumann??s puzzle[J]. Cognitive Computation, 2015, 7(3): 263-278.
[19] YE Z, KIM M K. Predicting electricity consumption in a building using an optimized back-propagation and Levenberg-Marquardt back-propagation neural network: case study of a shopping mall in China[J]. Sustainable Cities and Society, 2018, 42: 176-183.
[20] HUANG G B, SIEW C K. Extreme learning machine with randomly assigned RBF kernels[J]. International Journal of Information Technology, 2005, 11(1): 16-24.
[21] LU S Y, LU Z H, WANG S H, et al. Review of extreme learning machine[J]. Measurement & Control Technology, 2018, 37(10): 3-9.
陆思源, 陆志海, 王水花, 等. 极限学习机综述[J]. 测控技术, 2018, 37(10):3-9.
[22] XU R, LIANG X, QI J S, et al. Advances and trends in extreme learning machine[J]. Chinese Journal of Computers, 2019, 42(7): 1640-1670.
徐睿, 梁循, 齐金山, 等. 极限学习机前沿进展与趋势[J]. 计算机学报, 2019, 42(7): 1640-1670.
[23] LIU B, XIA S X, MENG F R, et al. Manifold regularized extreme learning machine[J]. Neural Computing and Appli-cations, 2016, 27(2): 255-269.
[24] YANG C, ZHU X, AHMAD Z, et al. Design of incremental echo state network using leave-one-out cross-validation[J]. IEEE Access, 2018, 6: 74874-74884.
[25] YANG M, MIN G, YANG W, et al. Software rejuvenation in cluster computing systems with dependency between nodes[J]. Computing, 2014, 96(6): 503-526.
[26] JIMENEZ L O, LANDGREBE D. High dimensional feature reduction via projection pursuit[C]//Proceedings of IGARSS??94-1994 IEEE International Geoscience and Remote Sensing Symposium, Pasadena, Aug 8-12, 1994. Piscataway: IEEE, 1994.
[27] HUANG R, HE M Y, YANG S J. A margin based feature extraction algorithm for the small sample size problem[J]. Chinese Journal of Computers, 2007, 30(7): 1173-1178.
黄睿, 何明一, 杨少军. 一种适用于小样本问题的基于边界的特征提取算法[J]. 计算机学报, 2007, 30(7): 1173-1178.
[28] BIAN Z Q. Pattern recognition[M]. Beijing: Tsinghua Uni-versity Press, 2000.
边肇祺. 模式识别[M]. 北京: 清华大学出版社, 2000.
[29] CAO J, ZHANG K, LUO M, et al. Extreme learning machine and adaptive sparse representation for image class-ification[J]. Neural Networks, 2016, 81: 91-102.