Bilateral Video Object Segmentation Using Dynamic Appearance Modeling and Higher-Order Potential

doi:10.3778/j.issn.1673-9418.1911014

Abstract

Abstract:

Aiming at the problems of poor quality and low time efficiency of video object segmentation in complex scenes, this paper proposes a novel bilateral video object segmentation using dynamic appearance modeling and higher-order potential, which is formulated the video object segmentation problem as binary labeling of Markov random field (MRF) based on bilateral grid cells. Firstly, this paper resamples each pixel in the video sequence with labeled keyframes into a higher-dimensional bilateral grid, which greatly reduces the video data to be processed. Secondly, this paper constructs a graph-cut optimization model using non-empty grid cells as the nodes of graph. The key is to construct the dynamic appearance model with confidence measurements, and to introduce a robust higher-order potential into the energy function. Finally, this paper uses the max-flow/min-cut algorithm to solve the global optimization problem, and binary label assignment of each pixel is achieved to obtain the high-quality video object segmentation. The experimental results on DAVIS 2016 and SegTrack v2 datasets show that with less user interaction, this method can not only obtain high-quality video object segmentation results for videos with complex scenes, but also significantly improve the time efficiency of video object segmentation.

Key words: video object segmentation, bilateral space, bilateral grid, confidence-based dynamic appearance model, higher-order energy potential

摘要：

针对复杂场景下视频目标分割质量不佳和时间效率低下的问题，提出了一种动态外观模型和高阶能量的双边视频目标分割方法，将视频目标分割转换为基于双边网格单元的马尔可夫随机场（MRF）模型求解问题。首先将带关键帧标记的视频序列映射至高维的双边网格，极大地减少待处理的数据。然后以非空网格单元作为图的结点并构建图割优化模型，其关键在于定义了具有置信度判别的动态外观模型，并在能量函数中引入鲁棒的高阶能量项。最后利用最大流/最小割算法进行全局优化求解，为视频像素点分配二值标签，最终获得高质量的视频目标分割结果。采用DAVIS 2016和SegTrack v2数据的实验结果表明，该方法在提供少量用户交互的情况下，不仅能在处理具有复杂场景的视频时获得理想的视频目标分割结果，而且还能显著提高视频目标分割的时间效率。

关键词: 视频目标分割, 双边空间, 双边网格, 置信动态外观模型, 高阶能量项

TIAN Ying, GUI Yan, XIONG Daming. Bilateral Video Object Segmentation Using Dynamic Appearance Modeling and Higher-Order Potential[J]. Journal of Frontiers of Computer Science and Technology, 2020, 14(12): 2108-2121.

田颖，桂彦，熊达铭. 动态外观模型和高阶能量的双边视频目标分割方法[J]. 计算机科学与探索, 2020, 14(12): 2108-2121.

References

[1] Papazoglou A, Ferrari V. Fast object segmentation in unconstrained video[C]//Proceedings of the 2013 IEEE International Conference on Computer Vision, Sydney, Dec 1-8, 2013. Piscataway: IEEE, 2013: 1777-1784.
[2] Wang W G, Shen J B, Porikli F. Saliency-aware geodesic video object segmentation[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, Jun 7-12, 2015. Piscataway: IEEE, 2015: 3395-3402.
[3] Jang W D, Lee C, Kim C S. Primary object segmentation in videos via alternate convex optimization of foreground and background distributions[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Piscataway: IEEE, 2016: 696-704.
[4] Brox T, Malik J. Object segmentation by long term analysis of point trajectories[C]//LNCS 6315: Proceedings of the 11th European Conference on Computer Vision, Heraklion, Sep 5-11, 2010. Berlin, Heidelberg: Springer, 2010: 282-295.
[5] Ochs P, Brox T. Object segmentation in video: a hierarchical variational approach for turning point trajectories into dense regions[C]//Proceedings of the 2011 IEEE International Conference on Computer Vision, Barcelona, Nov 6-13, 2011. Piscataway: IEEE, 2011: 1583-1590.
[6] Ochs P, Brox T. Higher order motion models and spectral clustering[C]//Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, Jun 16-21, 2012. Piscataway: IEEE, 2012: 614-621.
[7] Lezama J, Alahari K, Sivic J, et al. Track to the future: spatio-temporal video segmentation with long-range motion cues[C]//Proceedings of the 24th IEEE Conference on Computer Vision and Pattern Recognition, Colorado, Jun 20-25, 2011. Piscataway: IEEE, 2011: 3369-3376.
[8] Fragkiadaki K, Zhang G, Shi J B. Video segmentation by tracing discontinuities in a trajectory embedding[C]//Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, Jun 16-21, 2012. Piscataway: IEEE, 2012: 1846-1853.
[9] Li F X, Kim T, Humayun A, et al. Video segmentation by tracking many figure-ground segments[C]//Proceedings of the 2013 IEEE International Conference on Computer Vision, Sydney, Dec 1-8, 2013. Piscataway: IEEE, 2013: 2192-2199.
[10] Seguin G, Bojanowski P, Lajugie R, et al. Instance-level video segmentation from object tracks[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Piscataway: IEEE, 2016: 3678-3687.
[11] Lee Y J, Kim J, Grauman K. Key-segments for video object segmentation[C]//Proceedings of the 2011 IEEE International Conference on Computer Vision, Barcelona, Nov 6-13, 2011. Piscataway: IEEE, 2011: 1995-2002.
[12] Ma T Y, Latecki L J. Maximum weight cliques with mutex constraints for video object segmentation[C]//Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, Jun 16-21, 2012. Piscataway: IEEE, 2012: 670-677.
[13] Banica D, Agape A, Ion A, et al. Video object segmentation by salient segment chain composition[C]//Proceedings of the 2013 IEEE International Conference on Computer Vision Workshops, Sydney, Dec 1-8, 2013. Piscataway: IEEE, 2013: 283-290.
[14] Zhang D, Javed O, Shah M. Video object segmentation through spatially accurate and temporally dense extraction of primary object regions[C]//Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition, Portland, Jun 23-28, 2013. Piscataway: IEEE, 2013: 628-635.
[15] Faktor A, Irani M. Video segmentation by non-local consensus voting[C]//Proceedings of the British Machine Vision Conference, Nottingham, Sep 1-5, 2014. UK: BMVA, 2014: 21.
[16] Perazzi F, Wang O, Gross M, et al. Fully connected object proposals for video segmentation[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision, Santiago, Dec 7-13, 2015. Piscataway: IEEE, 2015: 3227-3234.
[17] Agarwala A, Hertzmann A, Salesin D, et al. Keyframe-based tracking for rotoscoping and animation[J]. ACM Transactions on Graphics, 2004, 23(3): 584-591.
[18] Bai X, Wang J, Simons D, et al. Video snapcut: robust video object cutout using localized classifiers[J]. ACM Trans-actions on Graphics, 2009, 28(3): 70.
[19] Bai X, Wang J, Sapiro G. Dynamic color flow: a motion-adaptive color model for object segmentation in video[C]//LNCS 6315: Proceedings of the 11th European Conference on Computer Vision, Heraklion, Sep 5-11, 2010. Berlin,Heidelberg: Springer, 2010: 617-630.
[20] Zhong F, Qin X Y, Peng Q S, et al. Discontinuity-aware video object cutout[J]. ACM Transactions on Graphics, 2012, 31(6): 175.
[21] Fan Q N, Zhong F, Lischinski D, et al. JumpCut: non-successive mask transfer and interpolation for video cutout[J]. ACM Transactions on Graphics, 2015, 34(6): 195.
[22] Lu Y, Bai X, Shapiro L G, et al. Coherent parametric contours for interactive video object segmentation[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Piscataway: IEEE, 2016: 642-650.
[23] Wang J, Bhat P, Colburn R A, et al. Interactive video cutout[J]. ACM Transactions on Graphics, 2005, 24(3): 585-594.
[24] Li Y, Sun J, Shum H Y. Video object cut and paste[J]. ACM Transactions on Graphics, 2005, 24(3): 595-600.
[25] Tsai D, Flagg M, Nakazawa A, et al. Motion coherent tracking using multi-label MRF optimization[J]. International Journal of Computer Vision, 2012, 100(2): 190-202.
[26] Jain S D, Grauman K. Supervoxel-consistent foreground propagation in video[C]//LNCS 8692: Proceedings of the 13th European Conference on Computer Vision, Zurich, Sep 6-12, 2014. Berlin, Heidelberg: Springer, 2014: 656-671.
[27] Nagaraja N S, Schmidt F R, Brox T. Video segmentation with just a few strokes[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision, Santiago, Dec 7-13, 2015. Piscataway: IEEE, 2015: 3235-3243.
[28] Tsai Y H, Yang M H, Black M J. Video segmentation via object flow[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, 2016. Piscataway: IEEE, 2016: 3899-3908.
[29] Marki N, Perazzi F, Wang O, et al. Bilateral space video segmentation[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, 2016. Piscataway: IEEE, 2016: 743-7511.
[30] Chen Y D, Hao C Y, Wu W, et al. Efficient frame-sequential label propagation for video object segmentation[J]. Multimedia Tools and Applications, 2018, 77(5): 6117-6133.
[31] Gui Y, Tang W, Zeng G. Gradient-constrained SLIC based fast video object segmentation[J]. Journal of Frontiers of Computer Science and Technology, 2019, 13(2): 285-299. 桂彦, 汤问, 曾光. 梯度约束SLIC的快速视频目标提取方法[J]. 计算机科学与探索, 2019, 13(2): 285-299.
[32] Boykov Y, Kolmogorov V. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2004, 26(9): 1124-1137.
[33] Perazzi F, Khoreva A, Benenson R, et al. Learning video object segmentation from static images[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, 2017. Piscataway: IEEE, 2017: 3491- 3500.
[34] Cheng J C, Tsai Y H, Wang S J, et al. Segflow: joint learning for video object segmentation and optical flow[C]//Proceedings of the 2017 International Conference on Computer Vision, Venice, Oct 22-29, 2017. Piscataway: IEEE, 2017: 686-695.
[35] Bao L C, Wu B Y, Liu W. CNN in MRF: video object segmentation via inference in a CNN-based higher-order spatio-temporal MRF[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, 2018. Piscataway: IEEE, 2018: 5977-5986.
[36] Wang Q, Zhang L, Bertinetto L, et al. Fast online object tracking and segmentation: a unifying approach[C]//Proc-eedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 1328-1338.
[37] Li Y, Sun J, Tang C K, et al. Lazy snapping[J]. ACM Transactions on Graphics, 2004, 23(3): 303-308.
[38] Rother C, Kolmogorov V, Blake A. “GrabCut”: interactive foreground extraction using iterated graph cuts[J]. ACM Transactions on Graphics, 2004, 23(3): 309-314.
[39] Yu H K, Zhou Y J, Qian H, et al. Loosecut: interactive image segmentation with loosely bounded boxes[C]//Proceedings of the 2017 IEEE International Conference on Image Processing, Beijing, Sep 17-20, 2017. Piscataway: IEEE, 2017: 3335-3339.
[40] Perazzi F, Pont-Tuset J, McWilliams B, et al. A benchmark dataset and evaluation methodology for video object segmentation[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Piscataway: IEEE, 2016: 724-732.
[41] Zhong F, Qin X Y, Peng Q S. Robust image segmentation against complex color distribution[J]. The Visual Computer, 2011, 27: 707-716.
[42] Grundmann M, Kwatra V, Han M, et al. Efficient hierarchical graph-based video segmentation[C]//Proceedings of the 23rd IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, Jun 13-18, 2010. Piscataway: IEEE, 2010: 2141-2148.
[43] Chang J, Wei D L, Fisher J W. A video representation using temporal superpixels[C]//Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition, Portland, Jun 23-28, 2013. Piscataway: IEEE, 2013: 2051-2058.