Melting Reduction Auto-Encoder

doi:10.3778/j.issn.1673-9418.2008046

Abstract

Abstract:

Auto-encoder (AE) is one of the simple and widely used unsupervised feature extraction algorithms of deep learning. Existing automatic encoders for image feature extraction remain some problems such as insufficient feature extraction and excessive model parameters, etc. Aiming at above problems, MRAE (melting reduction auto-encoder) is proposed for image feature extraction in this paper. Firstly, an “ablation network structure” is proposed in the algorithm. It can realize feature enrichment through feature cross fusion in the encoder and reduce feature loss and parameters of model by optimizing the decoding structure in the decoder. Secondly, a joint reconstruction loss function is designed. It calculates the reconstruction loss between feature layers to increase the relationship between feature layers and avoid the prematurity of the model. The experimental results show that the accuracy of the feature extracted by MRAE using different classifiers, such as SVM (support vector machine), K-means, and CART (classification and regression tree), is more than 97% on lung CT image datasets. The accuracy of the feature extracted by MRAE using fully connection is more than 90% on the CvD (cats vs. dogs) dataset.

Key words: auto-encoder (AE), feature extraction, melting reduction auto-encoder (MRAE), ablation network structure, loss function of joint reconstruction

摘要：

自动编码器（AE）是深度学习领域中一种结构简单且应用广泛的无监督特征提取算法。在图像特征提取方面，现有自动编码器普遍存在特征提取不充分、模型参数量较多等问题。针对上述问题，提出了一种用于图像特征提取的融减自动编码器（MRAE）。首先，在该算法中提出“融减网络结构”，该结构在编码器中通过特征交叉传递实现了特征融合，在解码器中通过优化解码结构降低了特征损失并减少了模型参数量；其次，设计一种联合重构损失函数，该函数通过计算特征层之间的重构损失，在加强特征层之间联系的同时可有效避免模型早熟。实验结果表明：在肺部CT图像数据集上，基于融减自动编码器所提取的特征使用支持向量机（SVM）、K-means和分类回归决策树（CART）等分类器，肺炎筛查准确率均在97%以上；在CvD数据集上，基于融减自动编码器所提取的特征使用全连接分类的准确率均在90%以上。

关键词: 自动编码器（AE）, 特征提取, 融减自动编码器（MRAE）, 融减网络结构, 联合重构损失函数

SUN Yu, WEI Benzheng, LIU Chuan, ZHANG Kuixing, CONG Jinyu. Melting Reduction Auto-Encoder[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(8): 1526-1533.

孙宇, 魏本征, 刘川, 张魁星, 丛金玉. 融减自动编码器[J]. 计算机科学与探索, 2021, 15(8): 1526-1533.

References

[1] RUMELHART D E, HINTON G E, WILLIAMS R J. Lear-ning representations by back-propagating errors[J]. Nature, 1986, 323(6088): 533-536.
[2] BOURLARD H, KAMP Y. Auto-association by multilayer perceptrons and singular value decomposition[J]. Biological Cybernetics, 1988, 59(4/5): 291-294.
[3] LAW A, GHOSH A. Multi-label classification using a cascade of stacked autoencoder and extreme learning machines[J]. Neurocomputing, 2019, 358: 222-234.
[4] AAMIR M, NAWI N M, WAHID F, et al. A deep contractive autoencoder for solving multiclass classification problems[J]. Evolutionary Intelligence, 2020: 1-15.
[5] LIU X Y, LIU J Q. Gait recognition method of underground coal mine personnel based on densely connected convolution network and stacked convolutional autoencoder[J]. Entropy, 2020, 22(6): 695.
[6] RIBEIRO M, LAZZARETTI A E, LOPES H S, et al. A study of deep convolutional auto-encoders for anomaly detection in videos[J]. Pattern Recognition Letters, 2018, 105: 13-22.
[7] LIU J R, WANG S P, YANG W Y, et al. Sparse autoencoder for social image understanding[J]. Neurocomputing, 2019, 369: 122-133.
[8] ZHANG Y, PENG H. Sample reconstruction with deep auto-encoder for one sample per person face recognition[J]. IET Computer Vision, 2017, 11(6): 471-478.
[9] LUCHNIKOV I, RYZHOV A, STAS P J C, et al. Variational autoencoder reconstruction of complex many-body physics[J]. Entropy, 2019, 21(11): 1091.
[10] SUN H, XU W, DENG C, et al. Multi-digit image synthesis using recurrent conditional variational autoencoder[C]//Pro-ceedings of the 2016 International Joint Conference on Neural Networks, Vancouver, Jul 24-29, 2016. Piscataway: IEEE, 2016: 375-380.
[11] MASCI J, MEIER U, CIRESAN D, et al. Stacked convolu-tional auto-encoders for hierarchical feature extraction[C]//Proceedings of the 2011 International Conference on Artificial Neural Networks, Espoo, Jun 14-17, 2011. Berlin, Heidelberg: Springer, 2011: 52-59.
[12] DU B, XIONG W, WU J, et al. Stacked convolutional denoising auto-encoders for feature representation[J]. IEEE Transactions on Cybernetics, 2017, 47(4): 1017-1027.
[13] ZHANG Q, YANG L T, CHEN Z, et al. Deep computation model for unsupervised feature learning on big data[J]. IEEE Transactions on Services Computing, 2016, 9(1): 161-171.
[14] NG A. Sparse autoencoder[Z]. CS294A Lecture Notes, 2011: 1-19.
[15] KINGMA D P, WELLING M. Auto-encoding variational Bayes[C]//Proceedings of the 2nd International Conference on Learning Representations, Banff, Apr 14-16, 2014.
[16] LIU W F, MA T Z, XIE Q S, et al. LMAE: a large margin auto-encoders for classification[J]. Signal Processing, 2017, 141: 137-143.
[17] RIFAI S, VINCENT P, MULLER X, et al. Contractive auto-encoders: explicit invariance during feature extraction[C]//Proceedings of the 28th International Conference on Machine Learning, Bellevue, Jun 28-Jul 2, 2011. New York: ACM, 2011: 833-840.
[18] XIE J, FANG Y, ZHU F, et al. Deep shape: deep learned shape descriptor for 3D shape matching and retrieval[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, Jun 8-10, 2015. Piscataway: IEEE, 2015: 1275-1283.
[19] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Confer-ence on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Piscataway: IEEE, 2016: 770-778.
[20] XU J, XIANG L, LIU Q, et al. Stacked sparse autoencoder (SSAE) for nuclei detection on breast cancer histopathology images[J]. IEEE Transactions on Medical Imaging, 2016, 35(1): 119-130.
[21] GLOROT X, BORDES A, BENGIO Y, et al. Deep sparse rectifier neural networks[J]. Journal of Machine Learning Research, 2011, 15: 315-323.
[22] HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 22-25, 2017. Piscataway: IEEE, 2017: 4700-4708.
[23] HAN Z, WEI B, MERCADO A, et al. Spine-GAN: semantic segmentation of multiple spinal structures[J]. Medical Image Analysis, 2018, 50: 23-35.
[24] VINCENT P, LAROCHELLE H, BENGIO Y, et al. Extracting and composing robust features with denoising autoencoders[C]//Proceedings of the 25th International Conference on Machine Learning, Helsinki, Jul 5-9, 2008. New York: ACM, 2008: 1096-1103.
[25] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv:1409. 1556, 2014.
[26] ZHANG K, LIU X, SHEN J, et al. Clinically applicable AI system for accurate diagnosis, quantitative measurements, and prognosis of COVID-19 pneumonia using computed tomography[J]. Cell, 2020, 181(6): 1423-1433.
[27] HAN Z, WEI B, HONG Y, et al. Accurate screening of COVID-19 using attention based deep 3D multiple instance learning[J]. IEEE Transactions on Medical Imaging, 2020, 39(8): 2584-2594.