RGB-D Image Saliency Detection via Background and Foreground Fusion

doi:10.3778/j.issn.1673-9418.1906021

Abstract

Abstract:

RGB-D image saliency detection refers to the addition of depth information in traditional 2D images to extract significant objects. However, for current saliency detection models, most of them focus on the saliency objects themselves, but ignore the background content. Therefore, this paper proposes a novel saliency detection model that takes depth information into consideration of both background and foreground to extract salient area. Firstly, the foreground noise is removed by the background measurement mechanism of the image boundary information and the background seed is selected from the boundary superpixels to calculate a background-based saliency map. Secondly, the input image is constructed into a graph, and the depth information is introduced into the graph. The foreground seeds are obtained by using clues such as color, depth and position, and the foreground-based saliency map is calculated. Finally, the background and foreground map are merged to obtain the initial saliency map, the cell optimization is performed, and the iterative propagation is performed to obtain the final saliency map. The comparison experiments are carried out on three RGB-D image saliency detection datasets LFSD, NJU-400 and NJU-2000. The experimental results show that the proposed method is effective and improves the accuracy.

Key words: RGB-D image saliency detection, foreground and background, salient image fusion, iterative propagation

摘要：

RGB-D图像显著性检测是指在传统的2D图像中附加深度信息从而提取显著对象，但是现有的显著性检测模型，大多数只关注显著物体本身，却忽略了背景信息。因此，提出了一个新颖的显著性检测模型，将深度信息同时考虑到背景和前景中提取出显著区域。首先，通过图像边界信息的背景测量机制来去除前景噪声并从边界超像素中选择背景种子，从而计算出基于背景的显著图；其次，将输入的图像构造成图，并将深度信息引入到图形结构中，利用颜色、深度、位置等线索获取前景种子，从而计算出基于前景的显著图；最后，将背景图和前景图融合获得初始显著图，再加以元胞优化，迭代传播后得到最终的显著图。在三个RGB-D图像显著性检测数据集LFSD、NJU-400、NJU-2000上进行对比实验，实验结果表明，该方法具备有效性，同时也提高了检测准确率。

关键词: RGB-D图像显著性检测, 前景和背景, 显著图像融合, 迭代传播

ZHAO Qiang, WANG Aiping, LIU Zhengyi. RGB-D Image Saliency Detection via Background and Foreground Fusion[J]. Journal of Frontiers of Computer Science and Technology, 2020, 14(7): 1232-1242.

赵强，王爱平，刘政怡. 背景与前景融合的RGB-D图像显著性检测[J]. 计算机科学与探索, 2020, 14(7): 1232-1242.

References

[1] Qian X T, Qian X M, Tan X L, et al. Sketch-based image retrieval by salient contour reinforcement[J]. IEEE Transac-tions on Multimedia, 2016, 18(8): 1604-1615.
[2] GaoY, Shi M J, Tao D C, et al. Database saliency for fast image retrieval[J]. IEEE Transactions on Multimedia, 2015, 17(3): 359-369.
[3] Humphrey K, Underwood G. Domain knowledge moderates the influence of visual saliency in scene recognition[J]. British Journal of Psychology, 2009, 100(2): 377-398.
[4] Zhang X D, Lv Y Y, Liao Y W, et al. Image saliency detection using regional covariance analysis[J]. Journal of Image and Graphics, 2016, 21(5): 605-615. 张旭东, 吕言言, 缪永伟, 等. 结合区域协方差分析的图像显著性检测[J]. 中国图象图形学报, 2016, 21(5): 605-615.
[5] Wang W H, Zhou J B, Gao S B, et al. Improved multi-scale saliency detection based on HSV space[J]. Computer Engin-eering and Science, 2017, 39(2): 364-370. 王文豪, 周静波, 高尚兵, 等. 基于HSV空间改进的多尺度显著性检测[J]. 计算机工程与科学, 2017, 39(2): 364-370.
[6] Itti L, Koch C, Niebur E. A model of saliency-based visual attention for rapid scene analysis[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1998, 20(11): 1254-1259.
[7] Achanta R, Hemami S S, Estrada F J, et al. Frequency-tuned salient region detection[C]//Proceedings of the 2009 IEEE International Conference on Computer Vision and Pattern Recognition, Miami, Jun 20-25, 2009. Washington: IEEE Computer Society, 2009: 1597-1604.
[8] Zhao H, An W S, Fu W G. Saliency detection algorithm of deep guidance[J]. Journal of Computer Applications, 2019, 39(1): 143-147. 赵恒, 安维胜, 付为刚. 深度导向显著性检测算法[J]. 计算机应用, 2019, 39(1): 143-147.
[9] Li H Y, Chen J, Lu H C, et al. CNN for saliency detection with low-level feature integration[J]. Neurocomputing, 2017, 226(C): 212-220.
[10] Desingh K, Krishna K M, Rajan D, et al. Depth really matters: improving visual salient region detection with depth[C]//Pro-ceedings of the 2013 British Machine Vision Conference, Bristol, Sep 9-13, 2013. Durham: BMVA Press, 2013: 98.
[11] Ciptadi A, Hermans T, Rehg J M. An in depth view of saliency[C]//Proceedings of the 2013 British Machine Vision Conference, Bristol, Sep 9-13, 2013. Durham: BMVA Press, 2013: 9-13.
[12] Chen L, Wu J G, Liu Z Y. Fusion of saliency map at image level and pixel level[J]. Journal of Frontiers of Computer Science and Technology, 2019, 13(12): 2130-2137. 陈蕾，吴建国，刘政怡. 图像级别和像素级别的显著图像的融合[J]. 计算机科学与探索, 2019, 13(12): 2130-2137.
[13] Yang C, Zhang L H, Lu H C, et al. Saliency detection via graph-based manifold ranking[C]//Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recogni-tion, Portland, Jun 23-28, 2013. Washington: IEEE Computer Society, 2013: 3166-3173.
[14] Goferman S, Zelnik-Manor L, Tal A. Context-aware saliency detection[J]. IEEE Transactions on Pattern Analysis and Ma-chine Intelligence, 2012, 34(10): 1915-1926.
[15] Wang H C, Zhang S L, Peng L. Salient region detection based on boundary information and color characteristics[J]. Computer Engineering and Applications, 2019, 55(3): 179-183. 王豪聪, 张松龙, 彭力. 融合边界信息和颜色特征的显著性区域检测[J]. 计算机工程与应用, 2019, 55(3): 179-183.
[16] Zhai J Y, Tu L Z, Zhuang Y. Saliency detection based on boundary prior and adaptive region merging[J]. Computer Engineering and Applications, 2018, 54(6): 178-182. 翟继友, 屠立忠, 庄严. 边界先验和自适应区域合并的显著性检测[J]. 计算机工程与应用, 2018, 54(6): 178-182.
[17] Liu Y N, Wu Q, Wei X. Saliency detection combined fore-ground with background based on manifold ranking[J]. Science Technology and Engineering, 2018, 18(18): 74-81. 刘亚宁, 吴清, 魏雪. 基于流行排序的前景背景显著性检测算法[J]. 科学技术与工程, 2018, 18(18): 74-81.
[18] Zhu W J, Liang S, Wei Y C, et al. Saliency optimization from robust background detection[C]//Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recogni-tion, Columbus, Jun 23-28, 2014. Washington: IEEE Com-puter Society, 2014: 2814-2821.
[19] Liu W, Xue H Y, Yu Z J, et al. RGB-D depth-map restora-tion using smooth depth neighborhood supports[J]. Journal of Electronic Imaging, 2015, 24(3): 033015.
[20] Achanta R, Shaji A, Smith K, et al. SLIC superpixels compared to state-of-the-art superpixel methods[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(11): 2274-2282.
[21] Cong R M, Lei J J, Zhang C Q, et al. Saliency detection for stereoscopic images based on depth confidence analysis and multiple cues fusion[J]. IEEE Signal Processing Letters, 2016, 23(6): 819-823.
[22] Alexe B, Deselaers T, Ferrari V. Measuring the objectness of image windows[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(11): 2189-2202.
[23] Von Neumann J. The general and logical theory of automata[M]//Jeffress L A. Cerebral Mechanisms in Behavior, 1951: 288-326.
[24] Li N Y, Ye J W, Ji Y, et al. Saliency detection on light field[C]//Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, Jun 23-28, 2014. Washington: IEEE Computer Society, 2014: 2806-2813.
[25] Ju R, Ge L, Geng W J, et al. Depth saliency based on aniso-tropic center-surround difference[C]//Proceedings of the 2014 IEEE International Conference on Image Processing, Paris, Oct 27-30, 2014. Piscataway: IEEE, 2014: 1115-1119.
[26] Ju R, Liu Y, Ren T W, et al. Depth-aware salient object detec-tion using anisotropic center-surround difference[J]. Signal Processing: Image Communication, 2015, 38: 115-126.
[27] Li H Y, Lu H C, Lin Z L, et al. Inner and inter label pro-pagation: salient object detection in the wild[J]. IEEE Transac-tions on Image Processing, 2015, 24(10): 3176-3186.
[28] Qin Y, Lu H C, Xu Y Q, et al. Saliency detection via cellular automata[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, Jun 7-12, 2015. Washington: IEEE Computer Society, 2015: 110-119.
[29] Peng H W, Li B, Ling H B, et al. Salient object detection via structured matrix decomposition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(4): 818-832.
[30] Feng D, Barnes N, You S D, et al. Local background enclosure for RGB-D salient object detection[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 2343-2350.
[31] Guo J F, Ren T W, Bei J. Salient object detection for RGB-D image via saliency evolution[C]//Proceedings of the 2016 IEEE International Conference on Multimedia and Expo, Seattle, Jul 11-15, 2016. Washington: IEEE Computer Society, 2016: 1-6.