Image Semantic Segmentation Method with Hierarchical Feature Fusion

doi:10.3778/j.issn.1673-9418.2005055

Abstract

Abstract:

When image feature information extraction is performed by using convolutional networks in the image semantic segmentation, the feature information between levels is not used effectively, resulting in image segmentation accuracy impaired. Therefore, an image semantic segmentation method of hierarchical feature fusion is proposed. The method uses the convolution structure to extract the low-level features at pixel level and the deep-level semantic features at image-level, the hidden features of different levels are delved step by step and the feature information hidden in the low-level features and deep-level semantic features are obtained fully. The upsampling is used to refine the low-level feature information and all the feature information is fused successively. Finally, the image semantic segmentation of hierarchical feature fusion is realized. In the aspect of experiment, through multiple decomposition experiments, the effect of this method is verified, the feature information extracted at different stages and the feature information extracted at different network depths, on the semantic segmentation results. At the same time, compared with three main methods on the image semantic segmentation PASCAL VOC 2012 dataset, the objective evaluation index and subjective effect performance results indicate that the proposed method is superior. It is verified that this method can effectively improve the accuracy of semantic segmentation.

Key words: convolutional neural networks (CNN), feature extraction, feature fusion, image semantic segmentation

摘要：

在图像语义分割中，利用卷积神经网络对图像信息进行特征提取时，针对卷积神经网络没有有效利用各层级间的特征信息而导致图像语义分割精度受损的问题，提出分级特征融合的图像语义分割方法。该方法利用卷积结构分级提取含有像素级的浅层低级特征和含有图像级的深层语义特征，进一步挖掘不同层级间的特征信息，充分获取浅层低级特征和深层语义特征中隐藏的特征信息，接着通过上采样操作细化浅层低级特征信息后对所有特征信息进行合并融合，最终实现分级特征融合的图像语义分割。在实验方面，通过多次分解实验验证了所提方法在不同阶段所提取的特征信息和不同网络深度时的特征信息对语义分割结果的影响。同时在公认的图像语义分割数据集PASCAL VOC 2012上，与3种主流方法进行实验对比，结果显示所提方法在客观评价指标和主观效果性能方面均存在优越性，从而验证了该方法可以有效地提升语义分割的精度。

关键词: 卷积神经网络（CNN）, 特征提取, 特征融合, 图像语义分割

ZHAO Xiaoqiang, XU Huiping. Image Semantic Segmentation Method with Hierarchical Feature Fusion[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(5): 949-957.

赵小强, 徐慧萍. 分级特征融合的图像语义分割[J]. 计算机科学与探索, 2021, 15(5): 949-957.

References

[1] WANG Y, ZHANG H J, HUANG H X. A survey of image semantic segmentation algorithms based on deep learning[J]. Application of Electronic Technique, 2019, 45(6): 23-27.
王宇, 张焕君, 黄海新. 基于深度学习的图像语义分割算法综述[J]. 电子技术应用, 2019, 45(6): 23-27.
[2] TIAN X, WANG L, DING Q. A survey of image semantic segmentation algorithms based on deep learning[J]. Journal of Software, 2019, 30(2): 440-468.
田萱, 王亮, 丁琪. 基于深度学习的图像语义分割方法综述[J]. 软件学报, 2019, 30(2): 440-468.
[3] WEI Y C, ZHAO Y. A review on image semantic segmenta-tion based on DCNN[J]. Journal of Beijing Jiaotong Univer-sity, 2016, 40(4): 82-91.
魏云超, 赵耀. 基于DCNN的图像语义分割综述[J]. 北京交通大学学报, 2016, 40(4): 82-91.
[4] AN Z, XU X P, YANG J H, et al. Design of segmented reality head-up display system based on image semantic segmenta-tion[J]. Acta Optica Sinica, 2018, 38(7): 0710004.
[5] XIONG Z Y, ZHANG G F, WANG J Q. Multi-scale feature extract for image semantic segmentation[J]. Journal of Cen-tral South University for Nationalities (Natural Science Edi-tion), 2017, 36(3): 118-124.
熊志勇, 张国丰, 王江晴. 基于多尺度特征提取的图像语义分割[J]. 中南民族大学学报(自然科学版), 2017, 36(3): 118-124.
[6] YI Z, CRIMINISI A, SHOTTON J, et al. Discriminative semantic segmentation of brain tissue in MR images[C]//LNCS 5762: Proceedings of the 2009 International Confer-ence on Medical Image Computing and Computer-Assisted Intervention. Berlin, Heidelberg: Springer, 2009: 558-565.
[7] WANG H Y, PAN D H, XIA D S. Fast implementation of two-dimensional Otsu adaptive threshold selection algorithm [J]. Acta Auto Matica Sinica, 2007(9): 968-971.
[8] YAMINI B, SABITHA R. Image steganalysis: adaptive color image segmentation using Otsu??s method[J]. Computational & Theoretical Nanoscience, 2017, 14(9): 4502-4507.
[9] HUANG Y, GUO L J, ZHANG R. Integration of global and local correntropy image segmentation algorithm[J]. Image and Graphics, 2018, 20(12): 1619-1628.
[10] CHOY S K, SHU Y L, YU K W, et al. Fuzzy model-based clustering and its application in image segmentation[J]. Pattern Recognition, 2017, 100(68): 141-157.
[11] COATES A, NG A Y. Learning feature representations with K-means[M]//MONTAVON G, ORR G B, MüLLER K R,eds. Neuralnetworks: Tricks of the Trade. Berlin, Heidelberg: Springer, 2012: 561-580.
[12] LIU S T, YIN F L. Image segmentation method based on graph cut and its new development[J]. Acta Auto Matica Sinica, 2012, 38(6): 911-922.
[13] HOJJATOLESLAMI S A, KITTLER J. Region growing: a new approach[J]. IEEE Transactions on Image Processing, 1998, 7(7): 1079-1084.
[14] BOYKOV Y Y, JOLLY M P. Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images[C]//Proceedings of the 8th International Conference on Computer Vision, Vancouver, Jul 7-14, 2001. Washing-ton: IEEE Computer Society, 2001: 105-112.
[15] HINTON G E, SALAKHUTDINOV R R. Reducing the dimensionality of data with neural networks[J]. Science, 2006, 313(5786): 504-507.
[16] RAZAVIAN A S, AZIZPOUR H, SULLIVAN J, et al. CNN features off-the-shelf: an astounding baseline for recogni-tion[C]//Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, Jun 23-28, 2014. Washington: IEEE Computer Society, 2014: 512-519.
[17] LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[J]. arXiv:1411.4038, 2014.
[18] NOH H, HONG S, HAN B. Learning deconvolution network for semantic segmentation[J]. arXiv:1505.04366, 2015.
[19] LIN G, MILAN A, SHEN C, et al. RefineNet: multi-path refinement networks with identity mappings for high-resolu-tion semantic segmentation[J]. arXiv:1611.06612, 2016.
[20] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. Sem-antic image segmentation with deep convolutional nets and fully connected CRFs[J]. arXiv:1412.7062, 2014.
[21] CHEN L C, PAPANDREOU G, SCHROFF F, et al. Rethink-ing atrous convolution for semantic image segmentation[J]. arXiv:1706.05587, 2017.
[22] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. arXiv:1606.00915, 2016.
[23] CHOLLET F. Xception: deep learning with depthwise se-parable convolutions[J]. arXiv:1610.02357, 2017.
[24] TIAN Z, HE T, SHEN C H, et al. Decoders matter for sem-antic segmentation: data-dependent decoding enables flexible feature aggregation[J]. arXiv:1903.02120, 2019.