Survey on Image Panoptic Segmentation Based on Deep Learning

doi:10.3778/j.issn.1673-9418.2304063

Abstract

Abstract: With the continuous development of deep learning and image segmentation, image panoptic segmentation has become a research hotspot in the field of computer vision, and many image panoptic segmentation methods have been proposed. This paper summarizes the research methods of image panoptic segmentation based on deep learning. Firstly, the research status of image panoptic segmentation at home and abroad is introduced, and the existing image panoptic segmentation methods are classified according to different optimization tasks in the network architecture, mainly including image panoptic segmentation optimized by feature extraction, image panoptic segmentation optimized by sub-task segmentation, image panoptic segmentation optimized by sub-task fusion, and other image panoptic segmentation. Secondly, 5 commonly used datasets, i.e. MS COCO, PASCAL VOC, Cityscapes, ADE20K and Mapillary Vistas, and 2 evaluation criteria, i.e. panoptic quality (PQ) and parsing covering (PC) in image panoptic segmentation are briefly introduced. And then, performance comparison of typical image panoptic segmentation methods has been conducted on different datasets. Thirdly, the application of image panoptic segmentation in medicine, autonomous driving, drones, agriculture, animal husbandry, military and other fields are listed. Finally, the deficiencies and challenges of existing methods in complex scene applications, real-time performance, and conflicts are pointed out, and the potential directions of image panoptic segmentation are discussed, including image panoptic segmentation based on a simple unified framework, real-time high-quality image panoptic segmentation, and image panoptic segmentation in complex application scenarios.

Key words: image panoptic segmentation, deep learning, feature extraction, sub-task segmentation, sub-task fusion

摘要： 随着深度学习与图像分割的不断发展，图像全景分割已经成为计算机视觉领域的一个研究热点，许多图像全景分割方法被提出。综述了基于深度学习的图像全景分割研究方法，首先介绍了图像全景分割国内外的研究现状，对已有图像全景分割的方法，根据网络架构优化任务的不同进行分类阐述，主要包括特征提取优化的图像全景分割、子任务分割优化的图像全景分割、子任务融合优化的图像全景分割、其他图像全景分割；其次简单介绍图像全景分割中常用的MS COCO、PASCAL VOC、Cityscapes、ADE20K和Mapillary Vistas五个数据集以及全景质量（PQ）和解析覆盖（PC）两种评价准则；然后对典型图像全景分割方法在不同数据集上进行了性能比较；接着列举了图像全景分割在医学、自动驾驶、无人机、农业、畜牧业、军事等领域的应用；最后指出了现有方法在复杂场景应用、实时性、冲突等方面存在的不足与挑战，并探讨了基于简单统一框架的图像全景分割、实时的高质量图像全景分割、复杂应用场景下图像全景分割等未来研究方向。

关键词: 图像全景分割, 深度学习, 特征提取, 子任务分割, 子任务融合

BI Yangyang, ZHENG Yuanfan, SHI Caijuan, ZHANG Kun, LIU Jian. Survey on Image Panoptic Segmentation Based on Deep Learning[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(11): 2605-2619.

毕阳阳, 郑远帆, 史彩娟, 张昆, 刘健. 基于深度学习的图像全景分割综述[J]. 计算机科学与探索, 2023, 17(11): 2605-2619.

References

[1] FORSYTH D A, MALIK J, FLECK M M, et al. Finding pictures of objects in large collections of images[C]//LNCS 1144: Proceedings of the 1996 International Workshop on Object Representation in Computer Vision. Berlin, Heidelberg: Springer, 1996: 335-360.
[2] MINAEE S, BOYKOV Y, PORIKLI F, et al. Image segmen-tation using deep learning: a survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 44(7): 3523-3542.
[3] LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, Jun 7-12, 2015. Washington: IEEE Computer Society, 2015: 3431-3440.
[4] 史彩娟, 陈厚儒, 张卫明, 等. 图像实例分割综述[C]//中国高科技产业化研究会智能信息处理产业化分会. 第十四届全国信号和智能信息处理与应用学术会议论文集, 北京, 2021.
SHI C J, CHEN H R, ZHANG W M, et al. Survey on image instance segmentation[C]//Intelligent Information Processing Industrialization Branch of China High Tech Industrialization Research Association. Proceedings of the 14th National Conference on Signal and Intelligent Information Processing and Application, Beijing, 2021.
[5] KIRILLOV A, HE K, GIRSHICK R, et al. Panoptic seg-mentation[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 15-20, 2019. Piscataway: IEEE, 2019: 9404-9413.
[6] ZHAO Z Q, ZHENG P, XU S, et al. Object detection with deep learning: a review[J]. IEEE Transactions on Neural Networks and Learning Systems, 2019, 30(11): 3212-3232.
[7] DVORNIK N, SHMELKOV K, MAIRAL J, et al. BlitzNet: a real-time deep network for scene understanding[C]//Pro-ceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Oct 22-29, 2017. Washington: IEEE Computer Society, 2017: 4174-4182.
[8] KIRILLOV A, GIRSHICK R, HE K, et al. Panoptic feature pyramid networks[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 15-20, 2019. Piscataway: IEEE, 2019: 6399-6408.
[9] LIN T Y, DOLLáR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jun 21-26, 2017. Washington: IEEE Computer Society, 2017: 936-944.
[10] HE K, GKIOXARI G, DOLLáR P, et al. Mask R-CNN[C]//Proceedings of the 2017 IEEE International Conference on Computer vision, Venice, Oct 22-29, 2017. Washington: IEEE Computer Society, 2017: 2961-2969.
[11] LI Q, ARNAB A, TORR P H S. Weakly-and semi-supervised panoptic segmentation[C]//LNCS 11219: Proceedings of the 15th European Conference on Computer Vision, Munich, Sep 8-14, 2018. Cham: Springer, 2018: 106-124.
[12] DE GEUS D, MELETIS P, DUBBELMAN G. Panoptic seg-mentation with a joint semantic and instance segmentation network[J]. arXiv:1809.02110, 2018.
[13] LI J, RAVENTOS A, BHARGAVA A, et al. Learning to fuse things and stuff[J]. arXiv:1812.01192, 2018.
[14] LI Y, CHEN X, ZHU Z, et al. Attention-guided unified network for panoptic segmentation[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 15-20, 2019. Piscataway: IEEE, 2019: 7026-7035.
[15] XIONG Y, LIAO R, ZHAO H, et al. UPSNet: a unified panoptic segmentation network[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 15-20, 2019. Piscataway: IEEE, 2019: 8818-8826.
[16] YANG T J, COLLINS M D, ZHU Y, et al. DeeperLab: single-shot image parser[J]. arXiv:1902.05093, 2019.
[17] LIU H, PENG C, YU C, et al. An end-to-end network for panoptic segmentation[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recogni-tion, Long Beach, Jun 15-20, 2019. Piscataway: IEEE, 2019: 6172-6181.
[18] DE GEUS D, MELETIS P, DUBBELMAN G. Fast panoptic segmentation network[J]. IEEE Robotics and Automation Letters, 2020, 5(2): 1742-1749.
[19] YANG Y, LI H, LI X, et al. SOGNet: scene overlap graph network for panoptic segmentation[C]//Proceedings of the 34th AAAI Conference on Artificial Intelligence, the 32nd Innovative Applications of Artificial Intelligence Conference, the 10th AAAI Symposium on Educational Advances in Artificial Intelligence, New York, Feb 7-12, 2020. Menlo Park: AAAI, 2020: 12637-12644.
[20] CHENG B, COLLINS M D, ZHU Y, et al. Panoptic-DeepLab: a simple, strong, and fast baseline for bottom-up panoptic segmentation[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recogni-tion, Seattle, Jun 13-19, 2020. Piscataway: IEEE, 2020: 12472-12482.
[21] BONDE U, ALCANTARILLA P F, LEUTENEGGER S. Towards bounding-box free panoptic segmentation[C]//LNCS 12544: Proceedings of the 42nd DAGM German Conference on Pattern Recognition, Tübingen, Sep 28-Oct 1, 2020. Cham: Springer, 2021: 316-330.
[22] WANG H, ZHU Y, GREEN B, et al. Axial-DeepLab: stand-alone axial-attention for panoptic segmentation[C]//LNCS 12349: Proceedings of the 16th European Conference on Computer Vision, Glasgow, Aug 23-28, 2020. Cham: Springer, 2020: 108-126.
[23] CHEN Y, LIN G, LI S, et al. BANet: bidirectional aggre-gation network with occlusion handling for panoptic seg-mentation[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 13-19, 2020. Piscataway: IEEE, 2020: 3792-3801.
[24] MOHAN R, VALADA A. EfficientPS: efficient panoptic segmentation[J]. International Journal of Computer Vision, 2021, 129(5): 1551-1579.
[25] WU Y, ZHANG G, GAO Y, et al. Bidirectional graph reasoning network for panoptic segmentation[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 13-19, 2020. Piscataway: IEEE, 2020: 9077-9086.
[26] HONG W, GUO Q, ZHANG W, et al. LPSNet: a lightweight solution for fast panoptic segmentation[C]//Proceedings of the 2021 IEEE Conference on Computer Vision and Pattern Recognition, Jun 19-25, 2021. Piscataway: IEEE, 2021: 16746-16754.
[27] LI Y, ZHAO H, QI X, et al. Fully convolutional networks for panoptic segmentation[C]//Proceedings of the 2021 IEEE Conference on Computer Vision and Pattern Recognition, Jun 19-25, 2021. Piscataway: IEEE, 2021: 214-223.
[28] WANG H, ZHU Y, ADAM H, et al. MaX-DeepLab: end-to-end panoptic segmentation with mask transformers[C]//Proceedings of the 2021 IEEE Conference on Computer Vision and Pattern Recognition, Jun 19-25, 2021. Piscataway: IEEE, 2021: 5463-5474.
[29] HWANG S, OH S W, KIM S J. Single-shot path integrated panoptic segmentation[C]//Proceedings of the 2022 IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, Jan 3-8, 2022. Piscataway: IEEE, 2022: 1939-1948.
[30] PORZI L, BULO S R, KONTSCHIEDER P. Improving pano-ptic segmentation at all scales[C]//Proceedings of the 2021 IEEE Conference on Computer Vision and Pattern Recogni-tion, Jun 19-25, 2021. Piscataway: IEEE, 2021: 7302-7311.
[31] ZHANG G, GAO Y, XU H, et al. Ada-Segment: automated multi-loss adaptation for panoptic segmentation[C]//Proceedings of the 35th AAAI Conference on Artificial Intelligence, the 33rd Conference on Innovative Applications of Artificial Intelligence, the 11th Symposium on Educational Advances in Artificial Intelligence, Feb 2-9, 2021. Menlo Park: AAAI, 2021: 3333-3341.
[32] HUANG J, GUAN D, XIAO A, et al. Cross-view regularization for domain adaptive panoptic segmentation[C]//Proceedings of the 2021 IEEE Conference on Computer Vision and Pattern Recognition, Jun 19-25, 2021. Piscataway: IEEE, 2021: 10133-10144.
[33] DE GEUS D, MELETIS P, LU C, et al. Part-aware panoptic segmentation[C]//Proceedings of the 2021 IEEE Conference on Computer Vision and Pattern Recognition, Jun 19-25, 2021. Piscataway: IEEE, 2021: 5485-5494.
[34] LI Z, WANG W, XIE E, et al. Panoptic SegFormer: delving deeper into panoptic segmentation with transformers[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, Jun 18-24, 2022. Piscataway: IEEE, 2022: 1270-1279.
[35] YU Q, WANG H, KIM D, et al. CMT-DeepLab: clustering mask transformers for panoptic segmentation[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, Jun 18-24, 2022. Piscat-away: IEEE, 2022: 2550-2560.
[36] GAO N, HE F, JIA J, et al. PanopticDepth: a unified framework for depth-aware panoptic segmentation[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, Jun 18-24, 2022. Piscataway: IEEE, 2022: 1622-1632.
[37] LI X, XU S, YANG Y, et al. Panoptic-PartFormer: learning a unified model for panoptic part segmentation[C]//LNCS 13687: Proceedings of the 17th European Conference on Computer Vision, Tel Aviv, Oct 23-27, 2022. Cham: Springer, 2022: 729-747.
[38] YU Q, WANG H, QIAO S, et al. k-means mask transformer[C]//LNCS 13689: Proceedings of the 17th European Confe-rence on Computer Vision, Tel Aviv, Oct 23-27, 2022. Cham: Springer, 2022: 288-307.
[39] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv:1409. 1556, 2014.
[40] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 770-778.
[41] HOWARD A G, ZHU M, CHEN B, et al. MobileNets: effi-cient convolutional neural networks for mobile vision appli-cations[J]. arXiv:1704.04861, 2017.
[42] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40(4): 834-848.
[43] GUO M H, XU T X, LIU J J, et al. Attention mechanisms in computer vision: a survey[J]. Computational Visual Media, 2022, 8(3): 331-368.
[44] REN S, HE K, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[C]//?Advances?in?Neural?Information?Processing?Systems?28,Montreal, Dec?7-12,?2015: 91-99.
[45] DAI J, HE K, SUN J. BoxSup: exploiting bounding boxes to supervise convolutional networks for semantic segmentation[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision, Santiago, Dec 7-13, 2015. Washington: IEEE Computer Society, 2015: 1635-1643.
[46] SUN B, KUEN J, LIN Z, et al. PRN: panoptic refinement network[C]//Proceedings of the 2023 IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, Jan 2-7, 2023. Piscataway: IEEE, 2023: 3952-3962.
[47] 金玲, 刘晓丽, 李鹏飞, 等. 遗传算法综述[J]. 科学中国人, 2015(27): 230.
JIN L, LIU X L, LI P F, et al. Survey of genetic algorithms[J]. Scientific Chinese, 2015(27): 230.
[48] 徐鹏斌, 瞿安国, 王坤峰, 等. 全景分割研究综述[J]. 自动化学报, 2021, 47(3): 549-568.
XU P B, QU A G, WANG K F, et al. A survey of panoptic segmentation methods[J]. Acta Automatica Sinica, 2021, 47(3): 549-568.
[49] ZHAO H, SHI J, QI X, et al. Pyramid scene parsing network[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 6230-6239.
[50] HU J, HUANG L, REN T, et al. You only segment once: towards real-time panoptic segmentation[J]. arXiv:2303.14651, 2023.
[51] CHANG S E, CHEN Y, YANG Y C, et al. SE-PSNet: silhouette-based enhancement feature for panoptic segmen-tation network[J]. Journal of Visual Communication and Image Representation, 2023, 90: 103736.
[52] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems?30,?Long?Beach,?Dec?4-9,?2017: 5998-6008.
[53] CHOWDHARY K R, CHOWDHARY K R. Natural language processing[J]. Fundamentals of Artificial Intelligence, 2020: 603-649.
[54] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: transformers for image recog-nition at scale[J]. arXiv:2010.11929, 2020.
[55] CARION N, MASSA F, SYNNAEVE G, et al. End-to-end object detection with transformers[C]//LNCS 12346: Procee-dings of the 16th European Conference on Computer Vision, Glasgow, Aug 23-28, 2020. Cham: Springer, 2020: 213-229.
[56] XIE E, WANG W, YU Z, et al. SegFormer: simple and efficient design for semantic segmentation with transformers[C]//Advances?in?Neural?Information?Processing?Systems?34,?Dec?6-14,?2021: 12077-12090.
[57] HAN K, WANG Y, CHEN H, et al. A survey on vision transformer[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(1): 87-110.
[58] KHAN S, NASEER M, HAYAT M, et al. Transformers in vision: a survey[J]. ACM Computing Surveys, 2022, 54: 1-41.
[59] LIU Z, LIN Y, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, Montreal, Oct 10-17, 2021. Piscataway: IEEE, 2021: 9992-10002.
[60] 杨俊闯, 赵超. K-Means聚类算法研究综述[J]. 计算机工程与应用, 2019, 55(23): 7-14.
YANG J C, ZHAO C. Survey on K-Means clustering algorithm[J]. Computer Engineering and Applications, 2019, 55(23): 7-14.
[61] SHEN Y, SONG K, TAN X, et al. HuggingGPT: solving AI tasks with ChatGPT and its friends in huggingface[J]. arXiv:2303.17580, 2023.
[62] ZHANG J, PU J, XUE J, et al. HiVeGPT: human-machine-augmented intelligent vehicles with generative pre-trained transformer[J]. IEEE Transactions on Intelligent Vehicles, 2023, 8(3): 2027-2033.
[63] WANG X, ZHANG X, CAO Y, et al. SegGPT: segmenting everything in context[J]. arXiv:2304.03284, 2023.
[64] LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO: common objects in context[C]//LNCS 8693: Proceedings of the 13th European Conference on Computer Vision, Zurich, Sep 6-12, 2014. Cham: Springer, 2014: 740-755.
[65] EVERINGHAM M, ESLAMI S M A, VAN GOOL L, et al. The Pascal visual object classes challenge: a retrospective[J]. International Journal of Computer Vision, 2015, 111: 98-136.
[66] CORDTS M, OMRAN M, RAMOS S, et al. The cityscapes dataset for semantic urban scene understanding[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 3213-3223.
[67] ZHOU B, ZHAO H, PUIG X, et al. Scene parsing through ADE20K dataset[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 5122-5130.
[68] NEUHOLD G, OLLMANN T, ROTA BULO S, et al. The Mapi-llary Vistas dataset for semantic understanding of street scenes[C]//Proceedings of the 2017 IEEE International Confe-rence on Computer Vision, Venice, Oct 22-29, 2017. Washington: IEEE Computer Society, 2017: 5000-5009.
[69] ZHANG D, SONG Y, LIU D, et al. Panoptic segmentation with an end-to-end cell R-CNN for pathology image analysis[C]//LNCS 11071: Proceedings of the 21st International Conference on Medical Image Computing and Computer Assisted Intervention, Granada, Sep 16-20, 2018. Cham: Springer, 2018: 237-244.
[70] YU X, LOU B, ZHANG D, et al. Deep attentive panoptic model for prostate cancer detection using biparametric MRI scans[C]//LNCS 12264: Proceedings of the 23rd International Conference on Medical Image Computing and Computer Assisted Intervention, Lima, Oct 4-8, 2020. Cham: Springer, 2020: 594-604.
[71] CHA J Y, YOON H I, YEO I S, et al. Panoptic segmentation on panoramic radiographs: deep learning-based segmentation of various structures including maxillary sinus and mandibular canal[J]. Journal of Clinical Medicine, 2021, 10(12): 2577.
[72] GINLEY B, LUCARELLI N, ZEE J, et al. Automated reference kidney histomorphometry using a panoptic segmentation neural network correlates to patient demographics and creatinine[C]//Proceedings of the Medical Imaging 2023: Digital and Computational Pathology, San Diego, Feb 19-24, 2023. San Francisco: SPIE, 2023: 458-462.
[73] GIRSHICK R, DONAHUE J, DARRELL T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation[C]//Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, Jun 23-28, 2014. Washington: IEEE Computer Society, 2014: 580-587.
[74] PETROVAI A, NEDEVSCHI S. Real-time panoptic seg-mentation with prototype masks for automated driving[C]//Proceedings of the 2020 IEEE Intelligent Vehicles Symposium, Las Vegas, Oct 19-Nov 13, 2020. Piscataway: IEEE, 2020: 1400-1406.
[75] CHEN H, DING L, YAO F, et al. Panoptic segmentation of UAV images with deformable convolution network and mask scoring[C]//Proceedings of the 12th International Conference on Graphics and Image Processing, Xi??an, Nov 13-15, 2020. San Francisco: SPIE, 2021: 312-321.
[76] DAI J, QI H, XIONG Y, et al. Deformable convolutional networks[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Oct 22-29, 2017. Washington: IEEE Computer Society, 2017: 764-773.
[77] SIDDIQUE A, TABB A, MEDEIROS H. Self-supervised learning for panoptic segmentation of multiple fruit flower species[J]. IEEE Robotics and Automation Letters, 2022, 7(4): 12387-12394.
[78] BRüNGER J, GENTZ M, TRAULSEN I, et al. Panoptic instance segmentation on pigs[J]. arXiv:2005.10499, 2020.
[79] SON J, LEE S. Hidden enemy visualization using fast panoptic segmentation on battlefields[C]//Proceedings of the 2021 IEEE International Conference on Big Data and Smart Com-puting, Jeju Island, Jan 17-20, 2021. Piscataway: IEEE, 2021: 291-294.
[80] BOLYA D, ZHOU C, XIAO F, et al. YOLACT: real-time instance segmentation[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Oct 27-Nov 2, 2019. Piscataway: IEEE, 2019: 9156-9165.