RGB-D Saliency Detection with Feature Enhancement and Progressive Decoding

doi:10.3778/j.issn.1673-9418.2412001

Abstract

Abstract: Aiming at the problems that most existing RGB-D saliency detection methods are difficult to effectively extract multi-scale features in the feature extraction process and the insufficient interaction of modal feature information in the decoding process leads to information loss, an RGB-D saliency detection method with feature enhancement and progressive decoding is proposed. Firstly, to address the difficulty of multi-scale feature extraction, which leads to low detection accuracy of the algorithm for multi-target and small-target objects, a multi-scale self-attentive feature enhancement module is constructed. Secondly, in order to fully utilize the advantages of the two modal features, a cross-modal feature fusion module is constructed to enhance the information interaction between different modal branches; and in order to solve the problem of information loss in the decoding process, a progressive weighted fusion decoder is proposed to fully integrate the feature information of different modalities and to refine the feature information. Finally, initial salient map optimization is achieved by supervised training of the layer-by-layer decoding results using a hybrid loss function. Through comparison experiments with 10 advanced mainstream methods on 5 public benchmark datasets, it is found that the method performs better on all 5 metrics. Experimental results show that the method achieves a more advanced performance by obtaining relatively accurate significant maps in a wide range of complex situations.

Key words: image processing, saliency detection, deep learning, feature enhancement, progressive decoding

摘要： 针对现有的大多数RGB-D显著性检测方法在特征提取过程中难以有效提取多尺度特征以及在解码过程中模态特征信息交互不充分导致信息丢失的问题，提出了一种特征增强和渐进式解码的RGB-D显著性检测方法。针对多尺度特征提取困难，导致算法对多目标、小目标物体检测精度低的问题，构建了多尺度自注意力特征增强模块；为了充分发挥两种模态特征的优势，构建了跨模态特征融合模块，以增强不同模态分支之间的信息交互；为了解决解码过程中信息易丢失的问题，提出了渐进式加权融合解码器，充分整合细化不同模态的特征信息；对逐层解码的结果采用混合损失函数监督训练，实现初始显著图优化。在5个公开基准数据集上与10种先进主流方法进行的对比实验表明，该方法在5个指标上均有较优表现。实验结果表明，该方法在多种复杂情况下都能获得相对精确的显著图，达到了较为先进的性能。

关键词: 图像处理, 显著性检测, 深度学习, 特征增强, 渐进式解码

HUA Chunjian, YAO Yetao, JIANG Yi, YU Jianfeng, CHEN Ying. RGB-D Saliency Detection with Feature Enhancement and Progressive Decoding[J]. Journal of Frontiers of Computer Science and Technology, 2025, 19(9): 2419-2429.

化春键, 姚烨涛, 蒋毅, 俞建峰, 陈莹. 特征增强和渐进式解码的RGB-D显著性检测[J]. 计算机科学与探索, 2025, 19(9): 2419-2429.

References

[1] 史彩娟, 张卫明, 陈厚儒, 等. 基于深度学习的显著性目标检测综述[J]. 计算机科学与探索, 2021, 15(2): 219-232.
SHI C J, ZHANG W M, CHEN H R, et al. Survey of salient object detection based on deep learning[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(2): 219-232.
[2] 赵宏伟, 霍东升, 王洁, 等. 基于显著性检测的害虫图像分类[J]. 吉林大学学报(工学版), 2021, 51(6): 2174-2181.
ZHAO H W, HUO D S, WANG J, et al. Image classification of insect pests based on saliency detection[J]. Journal of Jilin University (Engineering and Technology Edition), 2021, 51(6): 2174-2181.
[3] 钱晓亮, 曾银凤, 林生, 等. 融合自适应窗口显著性检测和改进超像素分割的高光谱异常检测[J]. 遥感学报, 2023, 27(12): 2748-2761.
QIAN X L, ZENG Y F, LIN S, et al. Hyperspectral anomaly detection via combining adaptive window saliency detection and improved superpixel segmentation[J]. National Remote Sensing Bulletin, 2023, 27(12): 2748-2761.
[4] LIU Z Y, SHI S, DUAN Q T, et al. Salient object detection for RGB-D image by single stream recurrent convolution neural network[J]. Neurocomputing, 2019, 363: 46-57.
[5] HAN J W, CHEN H, LIU N, et al. CNNs-based RGB-D saliency detection via cross-view transfer and multiview fusion[J]. IEEE Transactions on Cybernetics, 2018, 48(11): 3171-3183.
[6] WU Z W, ALLIBERT G, MERIAUDEAU F, et al. HiDAnet: RGB-D salient object detection via hierarchical depth awareness[J]. IEEE Transactions on Image Processing, 2023, 32: 2160-2173.
[7] 孟令兵, 袁梦雅, 时雪涵, 等. 跨模态融合和边界可变形卷积引导的RGB-D显著性目标检测[J]. 电子学报, 2023, 51(11): 3155-3166.
MENG L B, YUAN M Y, SHI X H, et al. RGB-D salient object detection based on cross-modal fusion and boundary deformable convolution guidance[J]. Acta Electronica Sinica, 2023, 51(11): 3155-3166.
[8] JU R, GE L, GENG W J, et al. Depth saliency based on anisotropic center-surround difference[C]//Proceedings of the 2014 IEEE International Conference on Image Processing. Piscataway: IEEE, 2014: 1115-1119.
[9] PENG H W, LI B, XIONG W H, et al. RGBD salient object detection: a benchmark and algorithms[C]//Proceedings of the 13th European Conference on Computer Vision. Cham: Springer, 2014: 92-109.
[10] NIU Y Z, GENG Y J, LI X Q, et al. Leveraging stereopsis for saliency analysis[C]//Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2012: 454-461.
[11] FAN D P, LIN Z, ZHANG Z, et al. Rethinking RGB-D salient object detection: models, data sets, and large-scale benchmarks[J]. IEEE Transactions on Neural Networks and Learning Systems, 2021, 32(5): 2075-2089.
[12] PANG Y W, ZHAO X Q, ZHANG L H, et al. CAVER: cross-modal view-mixed transformer for bi-modal salient object detection[J]. IEEE Transactions on Image Processing, 2023, 32: 892-904.
[13] LIU N, ZHANG N, WAN K Y, et al. Visual saliency transformer[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2021: 4702-4712.
[14] LIU Z Y, WANG Y, TU Z Z, et al. TriTransNet: RGB-D salient object detection with a triplet transformer embedding network[C]//Proceedings of the 29th ACM International Conference on Multimedia. New York: ACM, 2021: 4481-4490.
[15] LIU Z Y, TAN Y C, HE Q, et al. SwinNet: swin transformer drives edge-aware RGB-D and RGB-T salient object detection[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(7): 4486-4497.
[16] CONG R M, LIU H Y, ZHANG C, et al. Point-aware interaction and CNN-induced refinement network for RGB-D salient object detection[C]//Proceedings of the 31st ACM International Conference on Multimedia. New York: ACM, 2023: 406-416.
[17] ZHOU T, FU H Z, CHEN G, et al. Specificity-preserving RGB-D saliency detection[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2021: 4661-4671.
[18] JI W, LI J J, YU S, et al. Calibrated RGB-D salient object detection[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021: 9466-9476.
[19] LEE M, PARK C, CHO S, et al. SPSN: superpixel prototype sampling network for RGB-D salient object detection[C]//Proceedings of the 17th European Conference on Computer Vision. Cham: Springer, 2022: 630-647.
[20] LI G Y, LIU Z, CHEN M Y, et al. Hierarchical alternate interaction network for RGB-D salient object detection[J]. IEEE Transactions on Image Processing, 2021, 30: 3528-3542.