Survey of Salient Object Detection Based on Deep Learning

doi:10.3778/j.issn.1673-9418.2007074

Abstract

Abstract:

With the development of deep learning, salient object detection based on deep learning has become a research hotspot in the computer vision field. Firstly, existing salient object detection methods based on deep learning are introduced from three aspects, i.e. boundary/semantic enhancement, global/local combination and auxiliary network. As the same time, the saliency maps of these methods, qualitative analysis and comparison are given. Then the main datasets and main evaluation criteria for salient object detection based on deep learning are introduced in brief. Next the performance of salient object detection methods based on deep learning are compared on some datasets, including quantitative comparison, P-R curves and visual comparison. Finally, the shortcomings of the existing methods in complex background, small objects and real-time detection are pointed out, and the future development direction of salient object detection methods based on deep learning is explored, such as complex background, real-time, small object, weakly-supervised salient object detection and so on.

Key words: salient object detection, deep learning, visual saliency

摘要：

随着深度学习的不断发展，基于深度学习的显著性目标检测已经成为计算机视觉领域的一个研究热点。首先对现有的基于深度学习的显著性目标检测算法分别从边界/语义增强、全局/局部结合和辅助网络三个角度进行了分类介绍并给出了显著性图，同时对三种类型方法进行了定性分析比较；然后简单介绍了基于深度学习的显著性目标检测常用的数据集和评估准则；接着对所提基于深度学习的显著性目标检测方法在多个数据集上进行了性能比较，包括定量比较、P-R曲线和视觉比较；最后指出现有基于深度学习的显著性目标检测方法在复杂背景、小目标、实时性检测等方面的不足，并对基于深度学习的显著性目标检测的未来发展方向，如复杂背景、实时、小目标、弱监督等显著性目标检测进行了探讨。

关键词: 显著性目标检测, 深度学习, 视觉显著性

SHI Caijuan, ZHANG Weiming, CHEN Houru, GE Lulu. Survey of Salient Object Detection Based on Deep Learning[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(2): 219-232.

史彩娟, 张卫明, 陈厚儒, 葛录录. 基于深度学习的显著性目标检测综述[J]. 计算机科学与探索, 2021, 15(2): 219-232.

References

[1] ZHANG K, GUO Y R, WANG X S, et al. Channel-wise and feature-points reweights densenet for image classification[C]//Proceedings of the 2019 IEEE International Conference on Image Processing, Taiwan, China, Sep 22, 2019. Piscataway: IEEE, 2019: 410-414.
[2] BENJILALI W, GUICQUERO W, JACQUES L, et al. Hardware-friendly compressive imaging on the basis of random modulations & permutations for image acquisition and classification[C]//Proceedings of the 2019 IEEE International Conference on Image Processing, Taiwan, China, Sep 22, 2019. Piscataway: IEEE, 2019: 2085-2089.
[3] ZHANG Z Y, CUI Z, XU C Y, et al. Pattern-affinitive propagation across depth, surface normal and semantic segmentation [C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16, 2019. Washington: IEEE Computer Society, 2019: 4106-4115.
[4] DING H H, JIANG X D, SHUAI B, et al. Semantic correlation promoted shape-variant context for segmentation[C]//Procee-dings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16, 2019. Washington: IEEE Computer Society, 2019: 8885-8894.
[5] DI L, JI Y F, LISCHINSKI D, et al. Multi-scale context intertwining for semantic segmentation[C]//LNCS 11207: Proceedings of the 15th European Conference on Computer Vision, Munich, Sep 8, 2018. Berlin, Heidelberg: Springer, 2018: 622-638.
[6] GAO Y, WANG M, TAO D C. 3-D object retrieval and recognition with hypergraph analysis[J]. IEEE Transactions on Image Processing, 2012, 21(9): 4290-4303.
[7] HONG S, YOU T, KWAK S, et al. Online tracking by learning discriminative saliency map with convolutional neural network[C]//Proceedings of the 32nd International Conference on Machine Learning, Lille, Jul 6-11, 2015: 597-606.
[8] CHENG M M, ZHANG F L, MITRA N J, et al. Repfinder: finding approximately repeated scene elements for image editing[J]. ACM Transactions on Graphics, 2010, 29(4): 83.
[9] CRAYE C, FILLIAT D, GOUDOU J F, et al. Environment exploration for object-based visual saliency learning[C]//Proceedings of the 2016 IEEE International Conference on Robotics and Automation, Stockholm, May 16-21, 2016. Piscataway: IEEE, 2016: 2303-2309.
[10] CHENG M M, MITRA N J, HUANG X L, et al. Global contrast based salient region detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(3): 569-582.
[11] YANG C, ZHANG L H, LU H C, et al. Saliency detection via graph-based manifold ranking[C]//Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition, Portland, Jun 23-28, 2013. Washington: IEEE Computer Society, 2013: 3166-3173.
[12] JIANG Z, DAVIS L S. Submodular salient region detection[C]//Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition, Portland, Jun 23-28, 2013. Washington: IEEE Computer Society, 2013: 2043-2050.
[13] LIU J M, MENG W H. Review on single-stage object detection algorithm based on deep learning[J]. Aero Weaponry, 2020, 27(3): 44-53.
刘俊明, 孟卫华. 基于深度学习的单阶段目标检测算法研究综述[J]. 航空兵器, 2020, 27(3): 44-53.
[14] ZHENG W C, LI X W, LIU H Z. Survey of target detection algorithms based on deep learning[C]//China Computer Users Association Network Application Branch The 22nd Annual Network New Technology and Application Conference, Suzhou, Nov 8, 2018. Beijing: China Computer Users Association, 2018: 1-4.
郑伟成, 李学伟, 刘宏哲. 基于深度学习的目标检测算法综述[C]//中国计算机用户协会网络应用分会2018年第二十二届网络新技术与应用年会, 苏州, 2018-11-08. 北京：中国计算机用户协会, 2018: 1-4.
[15] ZHOU X L, CHEN X J, CHEN S Y, et al. Weakly supervised learning-based object detection: a survey[J]. Computer Science, 2019, 46(11): 49-57.
周小龙, 陈小佳, 陈胜勇, 等. 弱监督学习下的目标检测算法综述[J]. 计算机科学, 2019, 46(11): 49-57.
[16] ZHANG S D, YANG M, HU T. Salient object detection algorithm based on multi-feature fusion[J]. Journal of Frontiers of Computer Science and Technology, 2019, 13(5): 834-845.
张守东, 杨明, 胡太. 基于多特征融合的显著性目标检测算法[J]. 计算机科学与探索, 2019, 13(5): 834-845.
[17] SHI F F, ZHANG S L, PENG L. Deep convolution saliency detection combined with edge feature prior-based inspection[J]. Computer Engineering and Applications, 2020, 56(14): 199-206.
时斐斐, 张松龙, 彭力. 结合边缘特征先验引导的深度卷积显著性检测[J]. 计算机工程与应用, 2020, 56(14): 199-206.
[18] CHANG Z, DUAN X H, LU W C, et al. Multi-scale saliency detection based on Bayesian framework[J]. Computer Engineering and Applications, 2020, 56(11): 207-213.
常振, 段先华, 鲁文超, 等. 基于多尺度的贝叶斯模型显著性检测[J]. 计算机工程与应用, 2020, 56(11): 207-213.
[19] ZHANG Q, HUO Z, LIU Y, et al. Salient object detection employing a local tree-structured low-rank representation and foreground consistency[J]. Pattern Recognition, 2019, 92: 119-134.
[20] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[C]//Proceedings of the 3rd International Conference on Learning Representations, San Diego, May 7-9, 2015: 1-14.
[21] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE International Conference on Robotics and Automation, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 770-778.
[22] LEE G, TAI Y W, KIM J, et al. Deep saliency with encoded low level distance map and high level features[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 660-668.
[23] DAI J F, LI Y, HE K, et al. R-FCN: object detection via region-based fully convolutional networks[C]//Proceedings of the 30th International Conference on Neural Information Processing Systems, Barcelona, Dec 5, 2016. Massachusetts: MIT Press, 2016: 379-387.
[24] ZHANG P, WANG D, LU H, et al. Amulet: aggregating multi-level convolutional features for salient object detection[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Oct 22-29, 2017. Washington: IEEE Computer Society, 2017: 202-211.
[25] WANG T T, ZHANG L H, LU H C, et al. Kernelized subspace ranking for saliency detection[C]//LNCS 9912: Proceedings of the 14th European Conference on Computer Vision, Amsterdam, Oct 11-14, 2016. Berlin, Heidelberg: Springer, 2016: 450-466.
[26] LI G B, YU Y Z. Deep contrast learning for salient object detection[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 478-487.
[27] HOU Q B, CHENG M M, HU X W, et al. Deeply supervised salient object detection with short connections[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(4): 815-828.
[28] HOU Q B, LIU J J, CHENG M M, et al. Three birds one stone: a unified framework for salient object segmentation, edge detection and skeleton extraction[J]. arXiv:1803.09860, 2018.
[29] FENG M Y, LU H C, DING E, et al. Attentive feedback network for boundary-aware salient object detection[C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Washington: IEEE Computer Society, 2019: 1623-1632.
[30] QIN X B, ZHANG Z C, HUANG C Y, et al. BASNet: boundary-aware salient object detection[C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Washington: IEEE Computer Society, 2019: 7479-7489.
[31] WU Z, SU L, HUANG Q M, et al. Cascaded partial decoder for fast and accurate salient object detection[C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Washington: IEEE Computer Society, 2019: 3907-3916.
[32] LIU J J, HOU Q B, CHENG M M, et al. A simple pooling-based design for real-time salient object detection[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 3917-3926.
[33] LIU N, HAN J W, YANG M H, et al. PiCANet: learning pixel-wise contextual attention for saliency detection[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Washington: IEEE Computer Society, 2018: 3089-3098.
[34] CHEN S H, TAN X L, WANG B, et al. Reverse attention for salient object detection[C]//LNCS 11213: Proceedings of the 15th European Conference on Computer Vision, Munich, Sep 8-14, 2018. Berlin, Heidelberg: Springer, 2018: 236-252.
[35] ZHANG L, DAI J, LU H C, et al. A bi-directional message passing model for salient object detection[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Washington: IEEE Computer Society, 2018: 1741-1750.
[36] WANG T T, BORJI A, ZHANG L H, et al. A stagewise refinement model for detecting salient objects in images[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Oct 22-29, 2017. Washington: IEEE Computer Society, 2017: 4039-4048.
[37] WANG W G, ZHAO S Y, SHEN J B, et al. Salient object detection with pyramid attention and salient edges[C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Washington: IEEE Computer Society, 2019: 1448-1457.
[38] ZHAO T, WU X Q. Pyramid feature attention network for saliency detection[C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Washington: IEEE Computer Society, 2019: 3085-3094.
[39] LIU N, HAN J W. DHSNet: deep hierarchical saliency network for salient object detection[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 678-686.
[40] WANG T, ZHANG L, WANG S, et al. Detect globally, refine locally: a novel approach to saliency detection[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Piscataway: IEEE, 2018: 3127-3135.
[41] LUO Z M, MISHRA A K, ACHKAR A, et al. Non-local deep features for salient object detection[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 6593-6601.
[42] ZHANG X N, WANG T T, QI J Q, et al. Progressive attention guided recurrent network for salient object detection[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Piscataway: IEEE, 2018: 714-722.
[43] LI G, YU Y. Visual saliency based on multiscale deep features[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, Jun 7-13, 2015. Washington: IEEE Computer Society, 2015: 5455-5463.
[44] LI X, YANG F, CHENG H, et al. Contour knowledge transfer for salient object detection[C]//LNCS 11219: Proceedings of the 15th European Conference on Computer Vision, Munich, Sep 8-14, 2018. Berlin, Heidelberg: Springer, 2018: 370-385.
[45] ZHANG L, ZHANG J M, LIN Z, et al. CapSal: leveraging captioning to boost semantics for salient object detection[C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Washington: IEEE Computer Society, 2019: 6024-6033.
[46] WU R M, FENG M Y, GUAN W L, et al. A mutual learning method for salient object detection with intertwined multi-supervision[C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Washington: IEEE Computer Society, 2019: 8150-8159.
[47] LIU T, YUAN Z J, SUN J, et al. Learning to detect a salient object[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(2): 353-367.
[48] ACHANTA R, HEMAMI S S, ESTRADA F J, et al. Frequency-tuned salient region detection[C]//Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, Jun 20-25, 2009. Washington: IEEE Computer Society, 2009: 1597-1604.
[49] MOVAHEDI V, ELDER J H. Design and perceptual validation of performance measures for salient object segmentation [C]//Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, Jun 13-18, 2010. Washington: IEEE Computer Society, 2010: 49-56.
[50] LI Y, HOU X D, KOCH C, et al. The secrets of salient object segmentation[C]//Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, Jun 23-28, 2014. Washington: IEEE Computer Society, 2014: 280-287.
[51] WANG L J, LU H C, WANG Y F, et al. Learning to detect salient objects with image-level supervision[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 3796-3805.
[52] ALPERT S, GALUN M, BRANDT A, et al. Image segmentation by probabilistic bottom-up aggregation and cue integration[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(2): 315-327.
[53] SHI J, YAN Q, XU L, et al. Hierarchical image saliency detection on extended CSSD[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(4): 717-729.
[54] DENG J, DONG W, SOCHER R, et al. ImageNet: a large-scale hierarchical image database[C]//Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, Jun 20-25, 2009. Washington: IEEE Computer Society, 2009: 248-255.
[55] XIAO J X, HAYS J, EHINGER K A, et al. SUN database: large-scale scene recognition from abbey to zoo[C]//Proceedings of the 23rd IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, Jun 13-18, 2010. Washington: IEEE Computer Society, 2010: 3485-3492.
[56] EVERINGHAM M, VAN GOOL L, WILLIAMS C K I, et al. The pascal visual object classes (VOC) challenge[J]. International Journal of Computer Vision, 2010, 88(2): 303-338.
[57] MARGOLIN R, ZELNIK-MANOR L, TAL A, et al. How to evaluate foreground maps[C]//Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, Jun 23-28, 2014. Washington: IEEE Computer Society, 2014: 248-255.
[58] PERAZZI F, KR?HENBüHL P, PRITCH Y, et al. Saliency filters: contrast based filtering for salient region detection [C]//Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, Jun 16-21, 2012. Washington: IEEE Computer Society, 2012: 733-740.