Improved Siamese Adaptive Network and Multi-feature Fusion Tracking Algorithm

doi:10.3778/j.issn.1673-9418.2103044

Abstract

Abstract:

Aiming at the problem that tracking accuracy and tracking speed are difficult to balance in the current target tracking field. For example, a tracker based on correlation filtering can run at a very high speed, but the tracking accuracy is extremely low; a tracker based on deep learning can achieve high tracking accuracy, but the tracking speed is extremely low. On this basis, an improved Siamese adaptive network and multi-feature fusion target tracking algorithm are proposed. Firstly, the AlexNet network and the improved ResNet network are constructed on each branch of the Siamese network at the same time for feature extraction. Secondly, through end-to-end training at the same time, the tracking problem is decomposed into sub-problems of classifying each position label and returning to the bounding box. Finally, the shallow features and deep features are selected adaptively, and the target recognition and location are carried out based on multi-feature fusion. The proposed algorithm and some existing trackers are tested on the target tracking standard dataset. Experimental results show that the proposed algorithm can achieve high target tracking accuracy and success rate while ensuring tracking speed. At the same time, the algorithm has strong robustness in complex situations such as illumination changes, deformations, and background clutter.

Key words: object tracking, Siamese network, feature fusion, scale adaptation, ResNet network

摘要：

针对当前目标跟踪领域中跟踪精确度和跟踪速度难以平衡的问题，例如基于相关滤波实现的跟踪器能够以很高的速度运行，但跟踪准确性极低；基于深度学习实现的跟踪器能够实现较高的跟踪准确性，但跟踪速度较低。在此基础上，提出一种改进的Siamese自适应网络和多特征融合目标跟踪算法。首先在Siamese网络每个分支上同时构建AlexNet网络和改进的ResNet网络，用于特征提取。其次通过端到端的方式同时进行训练，将跟踪问题分解为分类每个位置标签和回归边界框子问题。最后对浅层特征和深层特征进行自适应选择，并基于多特征融合进行目标识别和定位。将提出的算法与现有的一些跟踪器在目标跟踪标准数据集上进行测试。实验结果表明, 提出的算法能够在确保跟踪速度的同时实现较高的跟踪精确度和成功率。同时，在光照变化、形变、背景杂波等复杂情况下，算法具有较强的鲁棒性。

关键词: 目标跟踪, Siamese网络, 特征融合, 尺度自适应, ResNet网络

CLC Number:

TP391

LI Rui, LIAN Jirong. Improved Siamese Adaptive Network and Multi-feature Fusion Tracking Algorithm[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(11): 2587-2595.

李睿, 连继荣. 改进的Siamese自适应网络和多特征融合跟踪算法[J]. 计算机科学与探索, 2022, 16(11): 2587-2595.

Figures/Tables 11

Fig.1 SiamFC network structure

Fig.2 Improved Siamese adaptive network structure

Fig.3 Residual unit structure

Fig.4 Improved residual unit structure

Fig.5 Tracking performance comparison

Fig.6 Accuracy graph and success rate graph

Fig.7 Accuracy graph and success rate graph drawn under influence of deformation

Fig.8 Accuracy graph and success rate graph drawn under influence of background clutter

Fig.9 Accuracy graph and success rate graph drawn under influence of occlusion

Table 1 Performance comparison between proposed algorithm and existing trackers

算法名称	成功率	精度	速度/（frame/s）
Ours	0.804	0.831	63.82
Struck	0.559	0.656	14.45
LOT	0.413	0.522	35.62
TLD	0.521	0.608	45.53
CT	0.348	0.406	20.59
SMS	0.185	0.337	89.57
MTT	0.445	0.475	49.83
CSK	0.443	0.545	94.66

Table 2 Algorithm internal comparison

名称	成功率	精度	速度/（frame/s）
N₁	0.586	0.642	80.35
N₂	0.713	0.686	68.61
N₃	0.786	0.813	62.54
F₁	0.467	0.516	98.96
F₂	0.424	0.498	25.31
F₃	0.658	0.579	56.32
M₁	0.864	0.882	48.69
M₂	0.753	0.806	92.73

References 35

[1]	陈晨, 邓赵红, 高艳丽, 等. 多模糊核融合的单目标跟踪算法[J]. 计算机科学与探索, 2020, 14(5): 848-860. DOI
	CHEN C, DENG Z H, GAO Y L, et al. Single target trac-king algorithm based on multi-fuzzy kernel fusion[J]. Jour-nal of Frontiers of Computer Science and Technology, 2020, 14(5): 848-860.
[2]	卢湖川, 李佩霞, 王栋. 目标跟踪算法综述[J]. 模式识别与人工智能, 2018, 31(1): 61-76. DOI
	LU H C, LI P X, WANG D. Visual object tracking: a survey[J]. Pattern Recognition and Artificial Intelligence, 2018, 31(1): 61-76. DOI
[3]	YANG H, SHAO L, ZHENG F, et al. Recent advances and trends in visual tracking: a review[J]. Neurocomputing, 2011, 74(18): 3823-3831. DOI URL
[4]	KALAL Z, MIKOLAJCZYK K, MATAS J. Tracking learning-detection[J]. IEEE Transactions on Pattern Analysis and Ma-chine Intelligence, 2012, 34(7): 1409-1422.
[5]	LI B, YAN J, WU W, et al. High performance visual tracking with siamese region proposal network[C]// Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Washington: IEEE Computer Society, 2018: 8971-8980.
[6]	BERTINETTO L, VALMADRE J, HENRIQUES J F, et al. Fully-convolutional siamese networks for object tracking[C]// LNCS 9914: Proceedings of the 14th European Conference on Computer Vision, Amsterdam, Oct 8-10 and 15-16, 2016. Cham: Springer, 2016: 850-865.
[7]	ZHU Z, WANG Q, LI B, et al. Distractor-aware siamese networks for visual object tracking[C]// LNCS 11213: Procee-dings of the 15th European Conference on Computer Vision, Munich, Sep 8-14, 2018. Cham: Springer, 2018: 103-119.
[8]	LI B, WU W, WANG Q, et al. SiamRPN++: evolution of siamese visual tracking with very deep networks[C]// Pro-ceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Pis-cataway: IEEE, 2019: 4282-4291.
[9]	GUO D, WANG J, CUI Y, et al. SiamCAR: siamese fully convolutional classification and regression for visual trac-king[C]// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 13-19, 2020. Piscataway: IEEE, 2020: 6268-6276.
[10]	SMEULDERS A W M, CHU D M, CUCCHIARA R, et al. Visual tracking: an experimental survey[J]. IEEE Transac-tions on Pattern Analysis and Machine Intelligence, 2013, 36(7): 1442-1468.
[11]	汪洪桥, 蔡艳宁, 孙富春, 等. 多尺度核方法的自适应序列学习及应用[J]. 模式识别与人工智能, 2011, 24(1): 72-81.
	WANG H Q, CAI Y N, SUN F C, et al. Adaptive sequence learning and applications for multi-scale kernel method[J]. Pattern Recognition and Artificial Intelligence, 2011, 24(1):72-81.
[12]	王玲, 王家沛, 王鹏, 等. 融合注意力机制的孪生网络目标跟踪算法研究[J]. 计算机工程与应用, 2021, 57(8): 169-174. DOI
	WANG L, WANG J P, WANG P, et al. Research on target tracking algorithm of twin network fusion attention mecha-nism[J]. Computer Engineering and Applications, 2021, 57(8): 169-174. DOI
[13]	单玉刚, 胡卫国. 尺度方向自适应视觉目标跟踪方法综述[J]. 计算机工程与应用, 2020, 56(9): 13-23. DOI
	SHAN Y G, HU W G. A survey of adaptive vision target tracking methods in scale and direction[J]. Computer Enginee-ring and Applications, 2020, 56(9): 13-23.
[14]	HENRIQUES J F, CASEIRO R, MARTINS P, et al. High-speed tracking with kernelized correlation filters[J]. IEEE Tran-sactions on Pattern Analysis and Machine Intelligence, 2015, 37(3): 583-596.
[15]	POSSEGGER H, MAUTHNER T, BISCHOF H. In defense of color-based model-free tracking[C]// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Re-cognition, Boston, Jun 7-12, 2015. Washington: IEEE Com-puter Society, 2015: 2113-2120.
[16]	VALMADRE J, BERTINETTO L, HENRIQUES J F, et al. End-to-end representation learning for correlation filter based tracking[C]// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 5000-5008.
[17]	GAO J Y, ZHANG T Z, XU C S. Graph convolutional trac-king[C]// Proceedings of the 2019 IEEE Conference on Com-puter Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 4649-4659.
[18]	ZHANG L, VARADARAJAN J, SUGANTHAN P N, et al. Robust visual tracking using oblique random forests[C]// Pro-ceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washing-ton: IEEE Computer Society, 2017: 5825-5834.
[19]	DANELLJAN M, BHAT G, KHAN F S, et al. ATOM: accurate tracking by overlap maximization[C]// Proceedings of the 2019 IEEE Conference on Computer Vision and Pat-tern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 4660-4669.
[20]	MA C, HUANG J B, YANG X K, et al. Hierarchical con-volutional features for visual tracking[C]// Proceedings of the 2015 IEEE International Conference on Computer Vision, Santiago, Dec 7-13, 2015. Washington: IEEE Computer So-ciety, 2015: 3074-3082.
[21]	BOLME D S, BEVERIDGE J R, DRAPER B A, et al. Visual object tracking using adaptive correlation filters[C]// Procee-dings of the 23rd IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, Jun 13-18, 2010. Washing-ton: IEEE Computer Society, 2010: 2544-2550.
[22]	LI F, YAO Y J, LI P H, et al. Integrating boundary and cen-ter correlation filters for visual tracking with aspect ratio variation[C]// Proceedings of the 2017 IEEE International Con-ference on Computer Vision Workshops, Venice, Oct 22-29, 2017. Washington: IEEE Computer Society, 2017: 2001-2009.
[23]	ZHANG J, MA S, SCLAROFF S. MEEM: robust tracking via multiple experts using entropy minimization[C]// LNCS 8694: Proceedings of the 13th European Conference on Com-puter Vision, Zurich, Sep 6-12, 2014. Cham: Springer, 2014: 188-203.
[24]	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. Image-Net classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6): 84-90. DOI URL
[25]	HE K M, ZHANG X Y, REN S Q, et al. Deep residual lear-ning for image recognition[C]// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition,Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer So-ciety, 2016: 770-778.
[26]	KRISTAN M, LEONARDIS A, MATAS J, et al. The visual object tracking vot2017 challenge results[C]// Proceedings of the 2017 IEEE International Conference on Computer Vi-sion, Venice, Oct 22-29, 2017. Washington: IEEE Computer Society, 2017: 1949-1972.
[27]	WU Y, LIM J, YANG M H. Object tracking benchmark[J]. IEEE Transactions on Pattern Analysis and Machine Intelli-gence, 2015, 37(9): 1834-1848.
[28]	RUSSAKOVSKY O, DENG J, SU H, et al. ImageNet large scale visual recognition challenge[J]. International Journal of Computer Vision, 2015, 115(3): 211-252. DOI URL
[29]	HARE S, GOLODETZ S, SAFFARI A, et al. Struck: struc-tured output tracking with kernels[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 65(5): 2096-2109.
[30]	ORON S, BAR-HILLEL A, LEVI D, et al. Locally orderless tracking[J]. International Journal of Computer Vision, 2015, 111(2): 213-228. DOI URL
[31]	KALAL Z, MATAS J, MIKOLAJCZYK K. P-N learning: bootstrapping binary classifiers by structural constraints[C]// Proceedings of the 23rd IEEE Conference on Computer Vi-sion and Pattern Recognition, San Francisco, Jun 13-18, 2010. Washington: IEEE Computer Society, 2010: 49-56.
[32]	ZHANG K H, ZHANG L, YANG M H. Real-time compre-ssive tracking[C]// LNCS 7574: Proceedings of the 12th Euro-pean Conference on Computer Vision, Florence, Oct 7-13, 2012. Berlin, Heidelberg: Springer, 2012: 864-877.
[33]	COLLINS R T. Mean-shift blob tracking through scale space[C]// Proceedings of the 2003 IEEE Computer Society Con-ference on Computer Vision and Pattern Recognition, Ma-dison, Jun 16-22, 2003. Washington: IEEE Computer So-ciety, 2003: 234-240.
[34]	ZHANG T Z, GHANEM B, LIU S, et al. Robust visual trac-king via multi-task sparse learning[C]// Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Re-cognition, Providence, Jun 16-21, 2012. Washington: IEEE Computer Society, 2012: 2042-2049.
[35]	HENRIQUES J F, RUI C, MARTINS P, et al. Exploiting the circulant structure of tracking-by-detection with kernels[C]// LNCS 7575: Proceedings of the 12th European Conference on Computer Vision, Florence, Oct 7-13, 2012. Berlin, Hei-delberg: Springer, 2012: 702-715.