Journal of Frontiers of Computer Science and Technology ›› 2023, Vol. 17 ›› Issue (11): 2605-2619.DOI: 10.3778/j.issn.1673-9418.2304063

• Frontiers·Surveys • Previous Articles     Next Articles

Survey on Image Panoptic Segmentation Based on Deep Learning

BI Yangyang, ZHENG Yuanfan, SHI Caijuan, ZHANG Kun, LIU Jian   

  1. 1. College of Artificial Intelligence, North China University of Science and Technology, Tangshan, Hebei 063210, China
    2. Hebei Key Laboratory of Industrial Intelligent Perception, Tangshan, Hebei 063210, China
  • Online:2023-11-01 Published:2023-11-01

基于深度学习的图像全景分割综述

毕阳阳,郑远帆,史彩娟,张昆,刘健   

  1. 1. 华北理工大学 人工智能学院,河北 唐山 063210
    2. 河北省工业智能感知重点实验室,河北 唐山 063210

Abstract: With the continuous development of deep learning and image segmentation, image panoptic segmentation has become a research hotspot in the field of computer vision, and many image panoptic segmentation methods have been proposed. This paper summarizes the research methods of image panoptic segmentation based on deep learning. Firstly, the research status of image panoptic segmentation at home and abroad is introduced, and the existing image panoptic segmentation methods are classified according to different optimization tasks in the network architecture, mainly including image panoptic segmentation optimized by feature extraction, image panoptic segmentation optimized by sub-task segmentation, image panoptic segmentation optimized by sub-task fusion, and other image panoptic segmentation. Secondly, 5 commonly used datasets, i.e. MS COCO, PASCAL VOC, Cityscapes, ADE20K and Mapillary Vistas,  and 2 evaluation criteria, i.e. panoptic quality (PQ) and parsing covering (PC) in image panoptic segmentation are briefly introduced. And then, performance comparison of typical image panoptic segmentation methods has been conducted on different datasets. Thirdly, the application of image panoptic segmentation in medicine, autonomous driving, drones, agriculture, animal husbandry, military and other fields are listed. Finally, the deficiencies and challenges of existing methods in complex scene applications, real-time performance, and conflicts are pointed out, and the potential directions of image panoptic segmentation are discussed, including image panoptic segmentation based on a simple unified framework, real-time high-quality image panoptic segmentation, and image panoptic segmentation in complex application scenarios.

Key words: image panoptic segmentation, deep learning, feature extraction, sub-task segmentation, sub-task fusion

摘要: 随着深度学习与图像分割的不断发展,图像全景分割已经成为计算机视觉领域的一个研究热点,许多图像全景分割方法被提出。综述了基于深度学习的图像全景分割研究方法,首先介绍了图像全景分割国内外的研究现状,对已有图像全景分割的方法,根据网络架构优化任务的不同进行分类阐述,主要包括特征提取优化的图像全景分割、子任务分割优化的图像全景分割、子任务融合优化的图像全景分割、其他图像全景分割;其次简单介绍图像全景分割中常用的MS COCO、PASCAL VOC、Cityscapes、ADE20K和Mapillary Vistas五个数据集以及全景质量(PQ)和解析覆盖(PC)两种评价准则;然后对典型图像全景分割方法在不同数据集上进行了性能比较;接着列举了图像全景分割在医学、自动驾驶、无人机、农业、畜牧业、军事等领域的应用;最后指出了现有方法在复杂场景应用、实时性、冲突等方面存在的不足与挑战,并探讨了基于简单统一框架的图像全景分割、实时的高质量图像全景分割、复杂应用场景下图像全景分割等未来研究方向。

关键词: 图像全景分割, 深度学习, 特征提取, 子任务分割, 子任务融合