Journal of Frontiers of Computer Science and Technology ›› 2022, Vol. 16 ›› Issue (12): 2718-2733.DOI: 10.3778/j.issn.1673-9418.2204041

• Surveys and Frontiers • Previous Articles     Next Articles

Survey of Deep Online Multi-object Tracking Algorithms

LIU Wenqiang, QIU Hangping(), LI Hang, YANG Li, LI Yang, MIAO Zhuang, LI Yi, ZHAO Xinxin   

  1. School of Command and Control Engineering, Army Engineering University of PLA, Nanjing 210007, China
  • Received:2022-04-13 Revised:2022-07-13 Online:2022-12-01 Published:2022-12-16
  • About author:LIU Wenqiang, born in 1996, M.S. candidate. His research interests include machine vision and multi-object tracking.
    QIU Hangping, born in 1965, Ph.D., professor. Her research interests include systems engineering and information retrieval.
    LI Hang, born in 1983, Ph.D., engineer. His research interests include computer vision and information fusion.
    YANG Li, born in 1981, M.S. candidate. His research interests include machine vision and object detection.
    LI Yang, born in 1984, Ph.D., associate professor. His research interests include artificial intelligence and deep learning.
    MIAO Zhuang, born in 1976, Ph.D., associate professor. His research interests include image and video processing.
    LI Yi,born in 1998, M.S. candidate. His research interests include artificial intelligence and image fusion.
    ZHAO Xinxin, born in 1996, M.S. candidate. Her research interests include machine learning and image retrieval.
  • Supported by:
    Natural Science Foundation of Jiangsu Province(BK20200581);Postdoctoral Science Foundation of China(2020M683754)

深度在线多目标跟踪算法综述

刘文强, 裘杭萍(), 李航, 杨利, 李阳, 苗壮, 李一, 赵昕昕   

  1. 陆军工程大学 指挥控制工程学院,南京 210007
  • 通讯作者: +E-mail: 13952004682@139.com
  • 作者简介:刘文强(1996—),男,江西上饶人,硕士研究生,主要研究方向为机器视觉、多目标跟踪。
    裘杭萍(1965—),女,浙江杭州人,博士,教授,主要研究方向为系统工程、信息检索。
    李航(1983—),男,江苏南京人,博士,工程师,主要研究方向为计算机视觉、信息融合。
    杨利(1981—),男,河北肃宁人,硕士研究生,主要研究方向为机器视觉、目标检测。
    李阳(1984—),男,河北廊坊人,博士,副教授,主要研究方向为人工智能、深度学习。
    苗壮(1976—),男,辽宁沈阳人,博士,副教授,主要研究方向为图像视频处理。
    李一(1998—),男,河北衡水人,硕士研究生,主要研究方向为人工智能、图像融合。
    赵昕昕(1996—),女,河南平顶山人,硕士研究生,主要研究方向为机器学习、图像检索。
  • 基金资助:
    江苏省自然科学基金(BK20200581);中国博士后科学基金(2020M683754)

Abstract:

Video multi-object tracking is a key task in the field of computer vision and has a wide application prospect in industry, commerce and military fields. At present, the rapid development of deep learning provides many solutions to solve the problem of multi-object tracking. However, the challenging problems such as mutation of target appearance, serious occlusion of target area, disappearance and appearance of target have not been completely solved. This paper focuses on online multi-object tracking algorithm based on deep learning, and summarizes the latest progress in this field. According to the three important modules of feature prediction, apparent feature extraction and data association, as will as the two frameworks of detection-based-tracking (DBT) and joint-detection-tracking (JDT), this paper divides deep online multi-object tracking algorithms into six sub-classes, and discusses the principles, advantages and disadvantages of different types of algorithms. Among them, the multi-stage design of the DBT algorithm has a clear structure and is easy to optimize, but multi-stage training may lead to sub-optimal solutions; the sub-modules of the JDT algorithm that integrates detection and tracking achieve faster inference speed, but there is a problem of collaborative training of each module. Currently, multi-target tracking begins to focus on long-term feature extraction of targets, occlusion target processing, association strategy improvement, and end-to-end framework design. Finally, combined with the existing algorithms, this paper summarizes urgent problems to be solved in deep online multi-object tracking and looks forward to possible research directions in the future.

Key words: online multi-object tracking, deep learning, feature extraction, data association

摘要:

视频多目标跟踪是计算机视觉领域的一个关键任务,在工业、商业及军事领域有着广泛的应用前景。目前,深度学习的快速发展为解决多目标跟踪问题提供了多种方案。然而,目标外观发生突变、目标区域被严重遮挡以及目标的消失和出现等挑战性的问题还未完全解决。重点关注基于深度学习的在线多目标跟踪算法,总结了该领域的最新进展,按照目标特征预测、表观特征提取和数据关联三个重要模块,依据基于检测跟踪(DBT)和联合检测跟踪(JDT)两个经典框架将深度在线多目标跟踪算法分为了六个小类,讨论不同类别算法的原理和优缺点。其中,DBT算法的多阶段设计结构清晰,容易优化,但多阶段的训练可能导致次优解;JDT算法融合检测和跟踪的子模块达到了更快的推理速度,但存在各模块协同训练的问题。目前,多目标跟踪开始关注目标的长期特征提取、遮挡目标处理、关联策略改进以及端到端框架的设计。最后,结合已有算法,总结了深度在线多目标跟踪亟待解决的问题并展望未来可能的研究方向。

关键词: 在线多目标跟踪, 深度学习, 特征提取, 数据关联

CLC Number: