Research and Development of Binocular Stereo Matching Algorithm

doi:10.3778/j.issn.1673-9418.2002030

Abstract

Abstract:

Binocular stereo vision has the advantages of low cost and wide applicability. It is extensively used in object recognition, object detection and has become the research focuses in the field of computer vision. Stereo matching is the critical process in the binocular stereo vision system, which plays an important role in the realization of three-dimensional reconstruction. The practical application of matching cost computation, cost aggregation, disparity computation/optimization and disparity refinement is described and the future trend of stereo matching is summarized and prospected based on the analysis of local, global, semi global and depth learning based stereo matching algorithms. Stereo matching will focus on solving the problem of mismatches in non-occluded and textureless regions. Deep learning algorithm for stereo matching has the advantages of high precision, and which is the future trend. Stereo matching algorithm will advance toward high precision, and real-time. Ideas and references are provided for the develop-ment of stereo matching of binocular vision technology.

Key words: binocular stereo vision, stereo matching, local stereo matching, global stereo matching, semi-global stereo matching, deep learning

摘要：

双目立体视觉技术具有成本低、适用性广的优点，在物体识别、目标检测等方面应用广泛，成为计算机视觉领域的研究热点。立体匹配是双目立体视觉技术中的核心算法，对实现物体三维重建具有重要作用。在分析局部、全局、半全局和基于深度学习的立体匹配算法研究现状的基础上，阐述了匹配代价、代价聚合、视差计算/优化、视差校正在立体匹配算法中的实际应用，总结及展望了立体匹配算法的发展趋势。立体匹配算法将着重解决无、弱纹理区域误匹配问题，深度学习算法用于立体匹配具有精度高的优点，是未来发展趋势，匹配算法将向高精度、实时性方向发展，为双目视觉技术中立体匹配算法的发展提供思路与参考。

关键词: 双目立体视觉, 立体匹配, 局部立体匹配, 全局立体匹配, 半全局立体匹配, 深度学习

ZHAO Chenyuan, LI Wenxin, ZHANG Qingxi. Research and Development of Binocular Stereo Matching Algorithm[J]. Journal of Frontiers of Computer Science and Technology, 2020, 14(7): 1104-1113.

赵晨园，李文新，张庆熙. 双目视觉的立体匹配算法研究进展[J]. 计算机科学与探索, 2020, 14(7): 1104-1113.

References

[1] Hu Y M, Li Y. A novel hierarchical object recognition algorithm based on saliency analysis[C]//Proceedings of the 37th Chinese Control Conference, Wuhan, Jul 25-27, 2018. Piscataway: IEEE, 2018: 6115-6120.
[2] Cao H, Wang R W, Wen X B, et al. A novel recognition algo-rithm in 3D point clouds based for on local spherical har-monics[C]//Proceedings of the 2019 IEEE International Con-ference on Mechatronics and Automation, Tianjin, Aug 4-7, 2019. Piscataway: IEEE, 2019: 1041-1046.
[3] Huang P C, Jiang J Y, Yang B. Research status and progress of binocular stereo vision[J]. Journal of Optical Instrument,2017, 40(4): 81-86.黄鹏程, 江剑宇, 杨波. 双目立体视觉的研究现状及进展[J]. 光学仪器, 2017, 40(4): 81-86.
[4] Shen T, Liu W B, Wang J. Distance measurement system based on binocular system vision[J]. Electronic Measurement Tech-nology, 2015, 38(4): 58-60.沈彤, 刘文波, 王京. 基于双目立体视觉的目标测距系统[J]. 电子测量技术, 2015, 38(4): 58-60.
[5] Sui J, Jin W Q. The realization and development of binocular stereo vision technology[J]. Application of Electronic Technique, 2004, 30(10): 4-6. 隋婧, 金伟其. 双目立体视觉技术的实现及其进展[J]. 电子技术应用, 2004, 30(10): 4-6.
[6] Binh P, Nguyen Q. A review of stereo-photogrammetry method for 3-D reconstruction in computer vision[C]//Proceedings of the 19th International Symposium on Communications and Information Technologies. Piscataway: IEEE, 2019: 138-143.
[7] Scharstein D, Szeliski R. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms[J]. International Journal of Computer Vision, 2002, 47: 7-42.
[8] Kitagawa M, Shimizu I. High accuracy local stereo matching using DoG scale map[C]//Proceedings of the 15th IAPR International Conference on Machine Vision Applications. Piscataway: IEEE, 2017: 258-261.
[9] Han P, Zhao M, Chen S. Fusion of texture, color and gradient information for stereo matching cost computation[C]//Proceed-ings of the 2nd International Conference on Image, Vision and Computing (ICIVC), Chengdu, Jun 2-4, 2017. Piscat-away: IEEE, 2017: 118-121.
[10] Chai Y, Yang F. Stereo matching algorithm based on joint matching cost and adaptive window[C]//Proceedings of the 2nd IEEE Advanced Information Management, Communicates, Electronic and Automation Control Conference, Chongqing, Oct 12-14, 2018. Piscataway: IEEE, 2018: 442-446.
[11] Hong G S, Kim B G. A local stereo matching algorithm based on weighted guided image filtering for improving the generation of depth range images[J]. Displays: Technology and Applications, 2017, 49: 80-87.
[12] Hamzah R A, Kadmin A F, Hamid M S, et al. Improvement of stereo matching algorithm for 3D surface reconstruction[J]. Signal Processing: Image Communication, 2018, 65: 165-172.
[13] Oussama Z, Mohammed R, Aouatif A, et al. A hierarchical stereo matching algorithm based on adaptive support region aggrega-tion method[J]. Pattern Recognition Letters, 2018, 112: 205-211.
[14] Ohta Y, Kanade T. Stereo by intra-and inter-scanline search using dynamic programming[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1985, PAMI-7(2): 139-154.
[15] Kolmogorov V, Zabih R. Computing visual correspondence with occlusions using graph cuts[C]//Proceedings of the 8th IEEE International Conference on Computer Vision, Vancouver, Jul 7-14, 2001. Piscataway: IEEE, 2001: 508-516.
[16] Yang Q, Wang L, Yang R, et al. Stereo matching with color-weighted correlation, hierachical belief propagation and occlu-sion handling[C]//Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision & Pattern Recogni-tion, New York, Jun 17-22, 2006. Piscataway: IEEE Computer Society, 2006: 1-8.
[17] Hallek M, Boukamcha H, Smach F, et al. Real time stereo matching using two step zero-mean SAD and dynamic pro-graming[C]//Proceedings of the 15th International Multi-Conference on Systems, Signals & Devices, Hammamet, Mar 19-22, 2018. Piscataway: IEEE, 2018: 1234-1240.
[18] Cheng F, Zhang H, Yuan D, et al. Stereo matching by using the global edge constraint[J]. Neurocomputing, 2014, 131:217-226.
[19] San T, War N. Stereo matching algorithm by hill-climbing segmentation[C]//Proceedings of the 6th IEEE Global Con-ference on Consumer Electronics, Nagoya, Oct 24-27, 2017.Piscataway: IEEE, 2017.
[20] Ma N, Men Y, Men C, et al. Segmentation-based stereo matching using combinatorial similarity measurement and adaptive support region[J]. International Journal for Light and Electron Optics, 2017, 137: 124-134.
[21] Wang X, Wang H, Su Y. Accurate belief propagation with parametric and non-parametric measure for stereo matching[J]. Optik, 2015, 126: 545-550.
[22] Lee S, Lee J H, Lim J, et al. Robust stereo matching using adaptive random walk with restart algorithm[J]. Image and Vision Computing, 2015, 37: 1-11.
[23] Tan X, Sun C, Pham T D. Stereo matching based on multi-direction polynomial model[J]. Signal Processing: Image Com-munication, 2016, 44: 44-56.
[24] Shahbazi M, Sohn G, Théau J. High-density stereo image matching using intrinsic curves[J]. ISPRS Journal of Photo-grammetry and Remote Sensing, 2018, 146: 373-388.
[25] Hirschmüller H. Accurate and efficient stereo processing by semi-global matching and mutual information[C]//Proceed-ings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2005: 807-814.
[26] Banz C, Hesselbarth S, Flatt H, et al. Real-time stereo vision system using semi-global matching disparity estimation: archi-tecture and FPGA-implementation[C]//Proceedings of the 2010 International Conference on Embedded Computer Systems, Samos, Jul 19-22, 2010. Piscataway: IEEE, 2010: 93-101.
[27] Li Y S, Zheng S Y, Wang X N, et al. An effificient photo-grammetric stereo matching method for high-resolution images[J]. Computers & Geosciences, 2016, 97: 58-66.
[28] Hernandez-Juarez D, Chacón A, Espinosa A, et al. Embedded real-time stereo estimation via semi-global matching on the GPU[J]. Procedia Computer Science, 2016, 80(C): 143-153.
[29] Wang E, Zhu Y, Peng L, et al. Stereo matching algorithm based on the combination of matching costs[C]//Proceedings of the 7th Annual IEEE International Conference on Cyber Technology in Automation, Control and Intelligent Systems, Honolulu, Jul 31-Aug 4, 2017. Piscataway: IEEE, 2017: 1001-1004.
[30] Chai Y, Yang F. Semi-global stereo matching algorithm based on minimum spanning tree[C]//Proceedings of the 2nd IEEE Advanced Information Management, Communicates, Elec-tronic and Automation Control Conference, Xi??an, May 25-27, 2018. Piscataway: IEEE, 2018: 2181-2185.
[31] Li Y, Li Z, Yang C, et al. High throughput hardware architec-ture for accurate semi-global matching[J]. Integration, the VLSI Journal, 2019, 65: 417-427.
[32] Rahnama O, Cavallari T, Golodetz S, et al. R3SGM: real-time raster-respecting semi-global matching for power-constrained systems[C]//Proceedings of the 2018 International Conference on Field Programmable Technology, Naha, Dec 10-14, 2018. Piscataway: IEEE, 2018: 105-112.
[33] Zbontar J, Lecun Y. Stereo matching by training a convolu-tional neural network to compare image patches[C]//Proceed-ings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, Jun 20-25, 2015. Piscataway: IEEE, 2015: 1-28.
[34] Chen L, Fan L, Chen J, et al. A full density stereo matching system based on the combination of CNNs and slanted-planes[J]. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2017, PP(99): 1-12.
[35] Yan T, Gan Y, Xia Z, et al. Segment-based disparity refine-ment with occlusion handling for stereo matching[J]. IEEE Transactions on Image Processing, 2019, 28(8): 3885-3897.
[36] Williem, In Kyu P. Deep self-guided cost aggregation for stereo matching[J]. Pattern Recognition Letters, 2018, 112:168-175.
[37] Lee Y, Kyung C. A memory- and accuracy-aware Gaussian parameter-based stereo matching using confidence measure[J]. IEEE Transactions on Pattern Analysisand Machine Intel-ligence, 2019, PP(99): 1-1.
[38] Hirschmüller H, Buder M, Ernst I. Memory effificient semi-global matching[J]. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2012, 3:371-376.
[39] Mayer N, Ilg E, Hausser P, et al. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Piscataway: IEEE, 2016: 4040-4048.
[40] Pang J, Sun W, Ren J S, et al. Cascade residual learning: atwo-stage convolutional neural network for stereo matching[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision Workshops, Venice, Oct 22-29, 2017.Piscataway: IEEE, 2017: 878-886.
[41] Kang J, Chen L, Deng F, et al. Context pyramidal network for stereo matching regularized by disparity gradients[J].ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 157: 201-215.
[42] Brandao P, Mazomenos E, Stoyanov D. Widening siamese architectures for stereo matching[J]. Pattern Recognition Letters, 2019, 120: 75-81.
[43] Nguyen T, Jeon J. Wide context learning network for stereo matching[J]. Signal Processing: Image Communication, 2019, 78: 263-273.
[44] Ran Q, Zhao W J, Feng J Q. Intrinsic color correction for stereo matching[J]. Computers & Graphics, 2019, 82: 22-31.
[45] Xue T, Owens A, Scharstein D, et al. Multi-frame stereo match-ing with edges, planes, and superpixels[J]. Image and Vision Computing, 2019, 91: 1-12.