Salient Object Detection with Feature Hybrid Enhancement and Multi-loss Fusion

doi:10.3778/j.issn.1673-9418.2104104

Journal of Frontiers of Computer Science and Technology ›› 2022, Vol. 16 ›› Issue (10): 2395-2404.DOI: 10.3778/j.issn.1673-9418.2104104

• Graphics and Image • Previous Articles Next Articles

Salient Object Detection with Feature Hybrid Enhancement and Multi-loss Fusion

LI Chunbiao, XIE Linbo⁺(), PENG Li

Engineering Research Center of Internet of Things Technology Applications (School of Internet of Things Engineering, Jiangnan University), Ministry of Education, Wuxi, Jiangsu 214122, China

Received:2021-04-20 Revised:2021-06-07 Online:2022-10-01 Published:2021-06-11
About author:LI Chunbiao, born in 1997, M.S. candidate. His research interests include salient object detection and deep learning.
XIE Linbo, born in 1973, Ph.D., professor, Ph.D. supervisor, member of CAA. His reearch interests include process modeling and control, intelligent detection and system safety.
PENG Li, born in 1967, Ph.D., professor, Ph.D. supervisor, member of CAAI and CCF. His research interests include visual Internet of things, action recognition and deep learning.
Supported by:
National Key Research and Development Program of China(2018YFD0400902);National Natural Science Foundation of China(61873112)

特征混合增强与多损失融合的显著性目标检测

李春标, 谢林柏⁺(), 彭力

物联网技术应用教育部工程研究中心（江南大学物联网工程学院）,江苏无锡 214122

通讯作者: + E-mail: xie_linbo@jiangnan.edu.cn
作者简介:李春标（1997—）,男,山东聊城人,硕士研究生,主要研究方向为显著性目标检测、深度学习。
谢林柏（1973—）,男,湖南永州人,博士,教授,博士生导师,CAA会员,主要研究方向为过程建模与控制、智能检测、系统安全性。
彭力（1967—）,男,河北唐山人,博士,教授,博士生导师,CAAI会员,CCF会员,主要研究方向为视觉物联网、行为识别、深度学习。
基金资助:
国家重点研发计划(2018YFD0400902);国家自然科学基金(61873112)

Abstract

Abstract:

To tackle the problem of missing features and poor regional consistency in existing salient object detec-tion algorithms, a salient object detection network which uses feature hybrid enhancement and multi-loss fusion based on fully convolutional neural network is proposed. The network includes a context-aware prediction module (CAPM) and a feature hybrid enhancement module (FHEM). First, the context-aware prediction module is used to extract the multi-scale feature information of the image, in which the spatial-aware module (SAM) is embedded to further extract the high-level semantic information of the image. Furthermore, the feature hybrid enhancement module is used to effectively integrate the global feature information and the detailed feature information generated by the prediction module, and the integrated feature is enhanced through embedded feature aggregation module (FAM). In addition, the multi-loss fusion method is used to supervise the network, which combines the binary cross-entropy (BCE) loss function, the structured similarity (SSIM) loss function and the proposed regional augmentation (RA) loss function. The network with the multi-loss fusion method can maintain the integrity of the foreground region and enhance the regional pixel consistency. The algorithm is verified on five image datasets with multiple salient objects and complex backgrounds. Experimental results demonstrate that the algorithm effectively improves the detection accuracy of saliency objects in complex scenes, and alleviates the problem of saliency map features missing and poor regional consistency.

Key words: convolutional neural network, salient object detection, feature hybrid enhancement, regional aug-mentation loss

摘要：

针对当前显著性目标检测算法存在的特征缺失和区域一致性差的问题,基于全卷积神经网络提出一种特征混合增强与多损失融合的显著性目标检测算法。该算法包含上下文感知预测模块（CAPM）和特征混合增强模块（FHEM）。首先利用上下文感知预测模块提取图像多尺度特征信息,并且在预测模块中嵌入空间感知模块（SAM）以进一步提取图像高层语义信息,然后利用特征混合增强模块对预测模块产生的全局特征信息和细节特征信息进行有效的整合,并利用特征聚合模块（FAM）对整合的信息进行特征增强。此外,提出了一种新的区域增强损失函数（RA）,并将此损失函数与已有的二进制交叉熵（BCE）损失函数、结构化相似度（SSIM）损失函数融合,以多损失融合的方式监督网络保持前景区域的完整性以及增强区域像素一致性。在五个包含多个显著性目标和复杂背景的图像数据集上对算法进行验证,实验结果表明,该算法有效地提高了复杂场景下显著性目标的检测精度,改善了显著图特征缺失与区域一致性差的问题。

关键词: 卷积神经网络, 显著性目标检测, 特征混合增强, 区域增强损失

CLC Number:

TP391.4

LI Chunbiao, XIE Linbo, PENG Li. Salient Object Detection with Feature Hybrid Enhancement and Multi-loss Fusion[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(10): 2395-2404.

李春标, 谢林柏, 彭力. 特征混合增强与多损失融合的显著性目标检测[J]. 计算机科学与探索, 2022, 16(10): 2395-2404.

Figures/Tables 11

Fig.1 Framework of salient object detection network

Fig.2 Spatial-aware module

Fig.3 Efficient channel attention module

Table 1 Performance comparison of different modules in algorithm

Configurations	$m a x F β$	MAE
Baseline U-Net	0.742	0.089
CAPM	0.784	0.068
CAPM+FHEM(CA)	0.773	0.072
CAPM+FHEM(ECA)	0.803	0.057

Table 1 Performance comparison of different modules in algorithm

Configurations	$m a x F β$	MAE
Baseline U-Net	0.742	0.089
CAPM	0.784	0.068
CAPM+FHEM(CA)	0.773	0.072
CAPM+FHEM(ECA)	0.803	0.057

Fig.4 Comparison of saliency detection results of different modules

Table 2 Performance comparison of different losses in algorithm

Configurations	$m a x F β$	MAE
CAPM+FHEM+ $l b c e$	0.789	0.069
CAPM+FHEM+ $l b s$	0.787	0.064
CAPM+FHEM+ $l b s i$	0.792	0.066
CAPM+FHEM+ $l b s r$	0.803	0.057

Table 2 Performance comparison of different losses in algorithm

Configurations	$m a x F β$	MAE
CAPM+FHEM+ $l b c e$	0.789	0.069
CAPM+FHEM+ $l b s$	0.787	0.064
CAPM+FHEM+ $l b s i$	0.792	0.066
CAPM+FHEM+ $l b s r$	0.803	0.057

Fig.5 Comparison of saliency detection results of different losses

Table 3 Performance comparison of RA loss with different coefficients in algorithm

$α$	$β$	$γ$	$m a x F β$	MAE
1.0	1.0	1.0	0.792	0.066
1.0	1.0	2.0	0.797	0.063
0.1	0.9	2.0	0.756	0.076
0.2	0.8	2.0	0.762	0.073
0.3	0.7	2.0	0.803	0.057
0.4	0.6	2.0	0.801	0.058
0.5	0.5	2.0	0.796	0.061

Table 3 Performance comparison of RA loss with different coefficients in algorithm

$α$	$β$	$γ$	$m a x F β$	MAE
1.0	1.0	1.0	0.792	0.066
1.0	1.0	2.0	0.797	0.063
0.1	0.9	2.0	0.756	0.076
0.2	0.8	2.0	0.762	0.073
0.3	0.7	2.0	0.803	0.057
0.4	0.6	2.0	0.801	0.058
0.5	0.5	2.0	0.796	0.061

Fig.6 Comparison of PR curves and F-measure curves of different algorithms

Table 4 Performance comparison of different algorithms on 5 datasets

Model	ECSSD		DUT-OMRON		DUTS-TE		HKU-IS		SOD
Model	$m a x F β$	MAE	$m a x F β$	MAE	$m a x F β$	MAE	$m a x F β$	MAE	$m a x F β$	MAE
Ours	0.942	0.036	0.803	0.057	0.860	0.046	0.928	0.031	0.857	0.102
CapsNet	0.887	0.052	0.703	0.063	0.799	0.048	0.880	0.039	—	—
RAS	0.921	0.056	0.786	0.062	0.831	0.059	0.913	0.045	0.850	0.124
Amulet	0.915	0.059	0.743	0.098	0.778	0.084	0.897	0.051	0.806	0.141
NLDF	0.905	0.063	0.753	0.080	0.812	0.066	0.902	0.048	0.841	0.124
UCF	0.911	0.078	0.734	0.132	0.771	0.117	0.886	0.074	0.803	0.164
RFCN	0.890	0.107	0.742	0.111	0.784	0.091	0.892	0.079	0.799	0.170
ELD	0.867	0.079	0.715	0.092	0.738	0.093	0.839	0.074	0.764	0.155
DCL	0.890	0.143	0.739	0.097	0.782	0.088	0.885	0.072	0.823	0.141
LEGS	0.827	0.118	0.669	0.133	0.655	0.138	0.766	0.119	0.734	0.196

Table 4 Performance comparison of different algorithms on 5 datasets

Model	ECSSD		DUT-OMRON		DUTS-TE		HKU-IS		SOD
Model	$m a x F β$	MAE	$m a x F β$	MAE	$m a x F β$	MAE	$m a x F β$	MAE	$m a x F β$	MAE
Ours	0.942	0.036	0.803	0.057	0.860	0.046	0.928	0.031	0.857	0.102
CapsNet	0.887	0.052	0.703	0.063	0.799	0.048	0.880	0.039	—	—
RAS	0.921	0.056	0.786	0.062	0.831	0.059	0.913	0.045	0.850	0.124
Amulet	0.915	0.059	0.743	0.098	0.778	0.084	0.897	0.051	0.806	0.141
NLDF	0.905	0.063	0.753	0.080	0.812	0.066	0.902	0.048	0.841	0.124
UCF	0.911	0.078	0.734	0.132	0.771	0.117	0.886	0.074	0.803	0.164
RFCN	0.890	0.107	0.742	0.111	0.784	0.091	0.892	0.079	0.799	0.170
ELD	0.867	0.079	0.715	0.092	0.738	0.093	0.839	0.074	0.764	0.155
DCL	0.890	0.143	0.739	0.097	0.782	0.088	0.885	0.072	0.823	0.141
LEGS	0.827	0.118	0.669	0.133	0.655	0.138	0.766	0.119	0.734	0.196

Fig.7 Comparison of saliency detection results of different algorithms

References 29

[1]	凌艳, 陈莹. 多尺度上下文信息增强的显著目标检测全卷积网络[J]. 计算机辅助设计与图形学学报, 2019, 31(11): 2007-2016.
	LING Y, CHEN Y. Salient object detection with multiscale context enhanced fully convolutional network[J]. Journal of Computer-Aided Design & Computer Graphics, 2019, 31(11): 2007-2016.
[2]	张守东, 杨明, 胡太. 基于多特征融合的显著性目标检测算法[J]. 计算机科学与探索, 2019, 13(5): 834-845. DOI
	ZHANG S D, YANG M, HU T. Salient object detection algorithm based on multi-feature fusion[J]. Journal of Fron-tiers of Computer Science and Technology, 2019, 13(5): 834-845.
[3]	OH S J, BENENSON R, KHOREVA A, et al. Exploiting sa-liency for object segmentation from image level labels[C]// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 22-25, 2017. Was-hington: IEEE Computer Society, 2017: 5038-5047.
[4]	GAO Y, WANG M, TAO D C, et al. 3-D object retrieval and recognition with hypergraph analysis[J]. IEEE Transactions on Image Processing, 2012, 21(9): 4290-4303. DOI PMID
[5]	REN Z X, GAO S H, CHIA L T, et al. Region-based saliency detection and its application in object recognition[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2014, 24(5): 769-779. DOI URL
[6]	ZHANG F, DU B, ZHANG L P. Saliency-guided unsupervised feature learning for scene classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2015, 53(4): 2175-2184. DOI URL
[7]	于明, 李博昭, 于洋, 等. 基于多图流形排序的图像显著性检测[J]. 自动化学报, 2019, 45(3): 577-592.
	YU M, LI B Z, YU Y, et al. Image saliency detection with multi-graph model and manifold ranking[J]. Acta Automatica Sinica, 2019, 45(3): 577-592.
[8]	常振, 段先华, 鲁文超, 等. 基于多尺度的贝叶斯模型显著性检测[J]. 计算机工程与应用, 2020, 56(11): 207-213. DOI
	CHANG Z, DUAN X H, LU W C, et al. Multi-scale saliency detection based on Bayesian framework[J]. Computer Eng-ineering and Applications, 2020, 56(11): 207-213.
[9]	时斐斐, 张松龙, 彭力. 结合边缘特征先验引导的深度卷积显著性检测[J]. 计算机工程与应用, 2020, 56(14): 199-206. DOI
	SHI F F, ZHANG S L, PENG L. Deep convolution saliency detection combined with edge feature prior-based inspec-tion[J]. Computer Engineering and Applications, 2020, 56(14): 199-206.
[10]	SCHARFENBERGER C, WONG A, FERGANI K, et al. Statistical textural distinctiveness for salient region detection in natural images[C]// Proceedings of the 2013 IEEE Con-ference on Computer Vision and Pattern Recognition, Port-land, Jun 23-28, 2013. Washington: IEEE Computer Society, 2013: 979-986.
[11]	KIM J H, HAN D Y, TAI Y W, et al. Salient region detec-tion via high-dimensional color transform[C]// Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, Jun 24-27, 2014. Washington: IEEE Computer Society, 2014: 883-890.
[12]	ZHU W J, LIANG S, WEI Y C, et al. Saliency optimization from robust background detection[C]// Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Re-cognition, Columbus, Jun 24-27, 2014. Washington: IEEE Computer Society, 2014: 2814-2821.
[13]	LIU Y, ZHANG Q, ZHANG D W, et al. Employing deep part-object relationships for salient object detection[C]// Pro-ceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Oct 27-Nov 2, 2019. Piscataway: IEEE, 2019: 1232-1241.
[14]	KUEN J, WANG Z H, WANG G. Recurrent attentional net-works for saliency detection[C]// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recogni-tion, Las Vegas, Jun 26-Jul 1, 2016. Washington: IEEE Computer Society, 2016: 3668-3677.
[15]	ZHANG P P, WANG D, LU H C, et al. Amulet: aggregating multi-level convolutional features for salient object detec-tion[C]// Proceedings of the 2017 IEEE International Con-ference on Computer Vision, Venice, Oct 22-29, 2017. Wa-shington: IEEE Computer Society, 2017: 202-211.
[16]	LUO Z M, MISHRA A K, ACHKAR A, et al. Non-local deep features for salient object detection[C]// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 6593-6601.
[17]	ZHANG P P, WANG D, LU H C, et al. Learning uncertain convolutional features for accurate saliency detection[C]// Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Oct 22-29, 2017. Washington: IEEE Computer Society, 2017: 212-221.
[18]	WANG L Z, WANG L J, LU H C, et al. Salient object detec-tion with recurrent fully convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(7): 1734-1746. DOI URL
[19]	WANG Z, SIMONCELLI E P, BOVIK A C. Multiscale struc-tural similarity for image quality assessment[C]// Proceedings of the 2003 IEEE Asilomar Conference on Signals, Pacific Grove, Nov 9-12, 2003. Washington: IEEE Computer Society, 2003.
[20]	FENG M Y, LU H C, DING E. Attentive feedback network for boundary-aware salient object detection[C]// Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 1623-1632.
[21]	QIN X B, ZHANG Z C, HUANG C Y, et al. BASNet: boun-dary-aware salient object detection[C]// Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Re-cognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 7471-7481.
[22]	RONNEBERGER O, FISCHER P, BROX T. U-Net: convo-lutional networks for biomedical image segmentation[C]// Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention, Mu-nich, Oct 5-9, 2015. Cham: Springer, 2015: 234-241.
[23]	ZHAO T, WU X Q. Pyramid feature attention network for saliency detection[C]// Proceedings of the 2019 IEEE Confe-rence on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 3080-3089.
[24]	ZHANG L H, WU J, WANG T T, et al. A multistage refine-ment network for salient object detection[J]. IEEE Transac-tions on Image Processing, 2020, 29: 3534-3545.
[25]	WANG Q L, WU B G, ZHU P F, et al. ECA-Net: efficient channel attention for deep convolutional neural networks[C]// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 13-19, 2020. Piscataway: IEEE, 2020: 11531-11539.
[26]	ACHANTA R, HEMAMI S, ESTRADA F, et al. Frequency-tuned salient region detection[C]// Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recogni-tion, Florida, Jun 20-25, 2009. Washington: IEEE Computer Society, 2009: 1597-1604.
[27]	LEE G, TAI Y W, KIM J. Deep saliency with encoded low level distance map and high level features[C]// Proceedings of the 2016 IEEE Conference on Computer Vision and Pat-tern Recognition, Las Vegas, Jun 26-Jul 1, 2016. Washing-ton: IEEE Computer Society, 2016: 660-668.
[28]	LI G B, YU Y Z. Deep contrast learning for salient object detection[C]// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 26-Jul 1, 2016. Washington: IEEE Computer Society, 2016: 478-487.
[29]	WANG L J, LU H C, RUAN X, et al. Deep networks for sa-liency detection via local estimation and global search[C]// Proceedings of the 2015 IEEE Conference on Computer Vi-sion and Pattern Recognition, Boston, Jun 7-12, 2015. Wa-shington: IEEE Computer Society, 2015: 3183-3192.

Salient Object Detection with Feature Hybrid Enhancement and Multi-loss Fusion

特征混合增强与多损失融合的显著性目标检测

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 11

References 29

Related Articles 15

Recommended Articles

Metrics

[1]	ZHU Bingyu, LIU Zhen, ZHANG Jingxiang. COVID-19 Detection Algorithm Combining Grad-CAM and Convolutional Neural Network [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(9): 2108-2120.
[2]	AN Fengping, LI Xiaowei, CAO Xiang. Medical Image Classification Algorithm Based on Weight Initialization-Sliding Window CNN [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(8): 1885-1897.
[3]	OU Yangliu, HE Xi, QU Shaojun. Fully Convolutional Neural Network with Attention Module for Semantic Segmentation [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(5): 1136-1145.
[4]	CHENG Weiyue, ZHANG Xueqin, LIN Kezheng, LI Ao. Deep Convolutional Neural Network Algorithm Fusing Global and Local Features [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(5): 1146-1154.
[5]	TONG Gan, HUANG Libo. Review of Winograd Fast Convolution Technique Research [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(5): 959-971.
[6]	PEI Lishen, ZHAO Xuezhuan. Survey of Collective Activity Recognition Based on Deep Learning [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(4): 775-790.
[7]	LU Zhongda, ZHANG Chunda, ZHANG Jiaqi, WANG Zifei, XU Junhua. Identification of Apple Leaf Disease Based on Dual Branch Network [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(4): 917-926.
[8]	LI Zhixin, CHEN Shengjia, ZHOU Tao, MA Huifang. Combining Cascaded Network and Adversarial Network for Object Detection [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(1): 217-230.
[9]	REN Longjie, SUN Ying, DING Weiping, JU Hengrong, CAO Jinxin. Multiple Lesions Detection of Fundus Images Based on CNN Algorithm Optimized by Single Population Frog-Leaping Algorithm [J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(9): 1762-1772.
[10]	ZHANG Mengqian, ZHANG Li. Coarse-to-Fine Two-Stage Convolutional Neural Network Algorithm [J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(8): 1501-1510.
[11]	FANG Junting, TAN Xiaoyang. Defect Detection of Metal Surface Based on Attention Cascade R-CNN [J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(7): 1245-1254.
[12]	NENG Wenpeng, LU Jun, ZHAO Caihong. Survey of Sleep Staging Based on Relational Induction Biases [J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(6): 1026-1037.
[13]	ZHAO Xiaoqiang, XU Huiping. Image Semantic Segmentation Method with Hierarchical Feature Fusion [J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(5): 949-957.
[14]	MA Dan, WAN Liang, CHENG Qiqin, SUN Zhiqiang. Research on Application of Attention-CNN in Malware Detection [J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(4): 670-681.
[15]	ZHANG Li, QIU Cunyue, ZHANG Kaixin, ZHANG Dabo, LUO Hao. Optimized Layered Convolutional Sub-health Recognition Algorithm of Improved Capsule Network [J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(4): 712-722.