Review of Research on 3D Reconstruction of Dynamic Scenes

doi:10.3778/j.issn.1673-9418.2305016

Abstract

Abstract: As static scene 3D reconstruction algorithms become more mature, dynamic scene 3D reconstruction has become a hot and challenging research topic in recent years. Existing static scene 3D reconstruction algorithms have good reconstruction results for stationary objects. However, when objects in the scene undergo deformation or relative motion, their reconstruction results are not ideal. Therefore, developing research on 3D reconstruction of dynamic scenes is essential. This paper first introduces the related concepts and basic knowledge of 3D reconstruction, as well as the research classification and current status of static and dynamic scene 3D reconstruction. Then, the latest research progress on dynamic scene 3D reconstruction is comprehensively summarized, and the reconstruction algorithms are classified into dynamic 3D reconstruction based on RGB data sources and dynamic 3D reconstruction based on RGB-D data sources. RGB data sources can be further divided into template based dynamic 3D reconstruction, non rigid motion recovery structure based dynamic 3D reconstruction, and learning based dynamic 3D reconstruction under RGB data sources. The RGB-D data source mainly summarizes dynamic 3D reconstruction based on learning, with typical examples introduced and compared. The applications of dynamic scene 3D reconstruction in medical, intelligent manufacturing, virtual reality and augmented reality, and transportation fields are also discussed. Finally, future research directions for dynamic scene 3D reconstruction are proposed, and an outlook on the research progress in this rapidly developing field is presented.

Key words: dynamic scene 3D reconstruction, template prior, motion recovery structure, deep learning

摘要： 随着静态场景三维重建算法的不断成熟，动态场景三维重建算法成为近年来的研究热点和研究难点。现有的静态场景三维重建算法对静止的对象有较好的重建效果，一旦场景中对象出现变形或者是相对运动，其重建效果不太理想，因此发展对动态场景的三维重建研究工作是相当重要的。简要介绍三维重建的相关概念及基本知识、静态场景三维重建和动态场景三维重建的研究分类及研究现状；全面总结了动态场景三维重建研究最新进展，将动态场景三维重建按照基于RGB数据源的动态三维重建和基于RGB-D数据源的动态三维重建进行分类，其中RGB数据源下又可划分为基于模板的动态三维重建、基于非刚性运动恢复结构的动态三维重建和RGB数据源下基于学习的动态三维重建，RGB-D数据源下主要总结归纳基于学习的动态三维重建，对各类典型重建算法进行了介绍和对比分析；介绍了动态场景三维重建在医学、智能制造、虚拟现实与增强现实、交通等领域的应用；提出了动态场景三维重建的未来研究方向，并对这个快速发展领域中的各个方向研究进行了展望。

关键词: 动态场景三维重建, 模板先验, 运动恢复结构, 深度学习

SUN Shuifa, TANG Yongheng, WANG Ben, DONG Fangmin, LI Xiaolong, CAI Jiacheng, WU Yirong. Review of Research on 3D Reconstruction of Dynamic Scenes[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(4): 831-860.

孙水发, 汤永恒, 王奔, 董方敏, 李小龙, 蔡嘉诚, 吴义熔. 动态场景的三维重建研究综述[J]. 计算机科学与探索, 2024, 18(4): 831-860.

References

[1] NEWCOMBE R A, IZADI S, HILLIGES O, et al. KinectFusion: real-time dense surface mapping and tracking[C]//Proceedings of the 2011 10th IEEE International Symposium on Mixed and Augmented Reality. Piscataway: IEEE, 2011: 127-136.
[2] WHELAN T, KAESS M, FALLON M, et al. Kintinuous: spatially extended kinectfusion[C]//Proceedings of the RSS Workshop on RGB-D: Advanced Reasoning with Depth Cameras, Sydney, 2012.
[3] CHOI S, ZHOU Q Y, KOLTUN V. Robust reconstruction of indoor scenes[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, Jun 7-12, 2015. Piscataway: IEEE, 2015: 5556-5565.
[4] NEWCOMBE R A, FOX D, SEITZ S M. DynamicFusion: reconstruction and tracking of non-rigid scenes in real-time[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, Jun 7-12, 2015. Piscataway: IEEE, 2015.
[5] INNMANN M, ZOLLHOFER M, NIE?NER M, et al. Volume-Deform: real-time volumetric non-rigid reconstruction[C]//Proceedings of the 14th European Conference on Computer Vision, Amsterdam, Oct 11-14, 2016. Cham: Springer, 2016: 362-379.
[6] WHELAN T, SALAS-MORENO R F, GLOCKER B, et al. ElasticFusion: real-time dense SLAM and light source estimation[J]. The International Journal of Robotics Research, 2016, 35(14): 1697-1716.
[7] YU T, GUO K, XU F, et al. BodyFusion: real-time capture of human motion and surface geometry using a single depth camera[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Oct 22-29, 2017. Piscataway: IEEE, 2017: 910-919.
[8] PRISACARIU V A, K?HLER O, GOLODETZ S, et al. InfiniTAM v3: a framework for large-scale 3D reconstruction with loop closure[J]. arXiv:1708.00783, 2017.
[9] DOU M, DAVIDSON P, FANELLO S R, et al. Motion2-Fusion: real-time volumetric performance capture[J]. ACM Transactions on Graphics, 2017, 36(6): 1-16.
[10] YU T, ZHENG Z, GUO K, et al. DoubleFusion: real-time capture of human performances with inner body shapes from a single depth sensor[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Piscataway: IEEE, 2018: 7287-7296.
[11] SLAVCHEVA M, BAUST M, ILIC S. SobolevFusion: 3D reconstruction of scenes undergoing free non-rigid motion[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Piscataway: IEEE, 2018: 2646-2655.
[12] ZHENG Z, YU T, LI H, et al. HybridFusion: real-time performance capture using a single depth sensor and sparse imus[C]//Proceedings of the 15th European Conference on Computer Vision, Munich, Sep 8-14, 2018. Cham: Springer, 2018: 384-400.
[13] XU L, SU Z, HAN L, et al. UnstructuredFusion: realtime 4D geometry and texture reconstruction using commercial RGBD cameras[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 42(10): 2508-2522.
[14] KLOKOV R, BOYER E, VERBEEK J. Discrete point flow networks for efficient point cloud generation[C]//Proceedings of the 16th European Conference on Computer Vision, Glasgow, Aug 23-28, 2020. Cham: Springer, 2020: 694-710.
[15] SU Z, XU L, ZHENG Z, et al. RobustFusion: human volumetric capture with data-driven visual cues using a RGBD camera[C]//Proceedings of the 16th European Conference on Computer Vision, Glasgow, Aug 23-28, 2020. Cham: Springer, 2020: 246-264.
[16] YANG G, SUN D, JAMPANI V, et al. LASR: learning articulated shape reconstruction from a monocular video[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 19-25, 2021. Piscataway: IEEE, 2021: 15980-15989.
[17] YANG G, SUN D, JAMPANI V, et al. ViSEr: video-specific surface embeddings for articulated 3D shape reconstruction[C]//Advances in Neural Information Processing Systems 34, 2021: 19326-19338.
[18] YU T, ZHENG Z, GUO K, et al. Function4D: real-time human volumetric capture from very sparse consumer RGBD sensors[C]//Proceedings of the 2021 IEEE Conference on Computer Vision and Pattern Recognition, Jun 19-25, 2021. Piscatawa: IEEE, 2021: 5746-5756.
[19] WILLIAMS F, GOJCIC Z, KHAMIS S, et al. Neural fields as learnable kernels for 3D reconstruction[C]//Proceedings of the 2022 IEEE Conference on Computer Vision and Pattern Recognition, New Orleans, Jun 19-24, 2022. Piscataway: IEEE, 2022: 18500-18510.
[20] YANG G, VO M, NEVEROVA N, et al. BANMo: building animatable 3D neural models from many casual videos[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, Jun 19-24, 2022. Piscataway: IEEE, 2022: 2863-2873.
[21] 戴波, 李雁飞, 安海洋, 等. 基于图像的危化品堆垛三维几何动态建模[J]. 控制工程, 2020, 27(1): 70-76.
DAI B, LI Y F, AN H Y, et al. Three dimensional geometric dynamic modeling of hazardous chemicals storage based on image[J]. Control Engineering of China, 2020, 27(1): 70-76.
[22] SATTLER F, BARNES S, CARRILLO PEREZ B J, et al. Real-time embedded reconstruction of dynamic objects for a 3D maritime situational awareness picture[C]//Proceedings of the European Workshop on Maritime Systems Resilience and Security 2022, Bremerhaven, Jun 20, 2022.
[23] 洪阳, 吴康, 李威, 等. 时空一致的动态纹理地图快速生成[J]. 计算机辅助设计与图形学学报, 2020, 32(5): 709-720.
HONG Y, WU K, LI W, et al. Fast generation of spatiotemporal- consistent dynamic atlas[J]. Journal of Computer-Aided Design & Computer Graphics, 2020, 32(5): 709-720.
[24] VO M, SHEIKH Y, NARASIMHAN S G. Spatiotemporal bundle adjustment for dynamic 3D human reconstruction in the wild[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 44(2): 1066-1080.
[25] 王刚, 江成浩, 刘世民, 等. 基于CT三维重建煤骨架结构模型的渗流过程动态模拟研究[J]. 煤炭学报, 2018, 43(5): 1390-1399.
WANG G, JIANG C H, LIU S M, et al. Dynamic simulation of seepage process based on CT 3D reconstruction of coal skeleton structure model[J]. Journal of China Coal Society, 2018, 43(5): 1390-1399.
[26] 单红梅, 吴岛, 张立斌, 等. 非接触式汽车轴距差动态检测方法[J]. 中南大学学报(自然科学版), 2017, 48(11): 2959-2965.
SHAN H M, WU D, ZHANG L B, et al. Dynamic detection method for non-contact vehicle wheelbase difference[J]. Journal of Central South University (Science and Technology), 2017, 48(11): 2959-2956.
[27] GOU R, CHEN G, PU X, et al. DR-Fusion: dynamic SLAM 3D reconstruction method of production workshop[C]//Proceedings of the 2022 5th International Conference on Pattern Recognition and Artificial Intelligence, Chendu, Aug 19-21, 2022. New York: IEEE, 2022: 556-561.
[28] LIN D, ZHANG A, GU J, et al. Detection of multipoint pulse waves and dynamic 3D pulse shape of the radial artery based on binocular vision theory[J]. Computer Methods and Programs in Biomedicine, 2018, 155: 61-73.
[29] SATTLER F, CARRILLO-PEREZ B, BARNES S, et al. Embedded 3D reconstruction of dynamic objects in real time for maritime situational awareness pictures[J]. The Visual Computer, 2023: 1-14.
[30] OHASHI T, IKEGAMI Y, NAKAMURA Y. Synergetic reconstruction from 2D pose and 3D motion for wide-space multi-person video motion capture in the wild[J]. Image and Vision Computing, 2020, 104: 104028.
[31] GUO K, TAYLOR J, FANELLO S, et al. TwinFusion: high framerate non-rigid fusion through fast correspondence tracking[C]//Proceedings of the 2018 International Conference on 3D Vision, Verona, Sep 5-8, 2018. New York: IEEE, 2018: 596-605.
[32] 陈立家, 王凯, 魏天明, 等. 基于动态流场数据的虚拟港口建模方法[J]. 交通运输工程学报, 2022, 22(2): 287-297.
CHEN L J, WANG K, WEI T M, et al. Virtual port modeling method based on dynamic fluid field data[J]. Journal of Traffic and Transportation Engineering, 2022, 22(2): 287-297.
[33] 张香玉, 金晖, 王丹. 动态场景下基于VR技术的短视频实时分割[J]. 计算机仿真, 2021, 38(8): 231-235.
ZHANG X Y, JIN H, WANG D. Real-time segmentation of short video in dynamic scene based on VR technology[J]. Computer Simulation, 2021, 38(8): 231-235.
[34] 韩海燕, 张静. 基于虚拟现实的三维动态场景重建[J]. 现代电子技术, 2018, 41(2): 170-173.
HAN H Y, ZHANG J. 3D dynamic scene reconstruction based on virtual reality[J]. Modern Electronics Technique, 2018, 41(2): 170-173.
[35] 付饶, 陈日清, 黄迎松, 等. 基于Delaunay剖分的心内膜表面动态三维重建算法[J]. 计算机应用研究, 2018, 35(10): 3113-3116.
FU R, CHEN R Q, HUANG Y S, et al. Delaunay based dynamic 3D endocardium surface reconstruction algorithm[J]. Application Research of Computers, 2018, 35(10): 3113-3116.
[36] 王星亮, 刘云鹏, 徐志庆, 等. 基于静态磁共振图像构建髌股关节准动态三维运动模型[J]. 中国临床解剖学杂志, 2017, 35(1): 69-73.
WANG X L, LIU Y P, XU Z Q, et al. Three- dimensional reconstruction of subject- specific dynamic patella of femoral joint using static magnetic resonance based methodology[J]. Chinese Journal of Clinical Anatomy, 2017, 35(1): 69-73.
[37] REDDY N D, VO M, NARASIMHAN S G. CarFusion: combining point tracking and part detection for dynamic 3D reconstruction of vehicles[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Piscataway: IEEE, 2018: 1906-1915.
[38] MAHAYUDDIN Z R, SAIF A S. A comprehensive review towards segmentation and detection of cancer cell and tumor for dynamic 3D reconstruction[J]. Asia-Pacific Journal of Information Technology and Multimedia, 2020, 9(1): 28-39.
[39] NARASIMHAN S G, TAMBURO R, REDDY D. Dynamic 3D reconstruction of vehicles for safer intersections[R]. Washington: University Transportation Center. Technologies for Safe and Efficient Transportation, 2018.
[40] 何爱军, 郑昌琼, 汪天富, 等. 组织多普勒超声心脏图像的动态三维重建[J]. 生物医学工程学杂志, 2005, 22(3): 570-574.
HE A J, ZHENG C Q, WANG T F, et al. Dynamic three- dimensional reconstruction of tissue doppler ultrasound heart images[J]. Journal of Biomedical Engineering, 2005, 22(3): 570-574.
[41] GUAN J, YANG X, LEE V C S, et al. Full field-of-view pavement stereo reconstruction under dynamic traffic conditions: incorporating height-adaptive vehicle detection and multi-view occlusion optimization[J]. Automation in Construction, 2022, 144: 104615.
[42] XUN F X, CANAVESE F, XU H W, et al. Dynamic 3D reconstruction of thoracic cage and abdomen in children and adolescents with scoliosis: preliminary results of optical reflective motion analysis assessment[J]. Journal of Pediatric Orthopaedics, 2020, 40(4): 196-202.
[43] KOREHISA S, IKEDA T, OKANO S, et al. A novel histological examination with dynamic three-dimensional reconstruction from multiple immunohistochemically stained sections of a PD-L1-positive colon cancer[J]. Histopathology, 2018, 72(4): 697-703.
[44] 卢林鹏, 关柏良, 林淑金. 面向点云三维重建的空间感知对抗神经网络[J]. 计算机工程与科学, 2022, 44(7): 1247-1255.
LU L P, GUAN B L, LIN S J. A space-aware adversarial neural network for 3D reconstruction of point cloud[J]. Computer Engineering & Science, 2022, 44(7): 1247-1255.
[45] TIAN X, LIU R, WANG Z, et al. High quality 3D reconstruction based on fusion of polarization imaging and binocular stereo vision[J]. Information Fusion, 2022, 77: 19-28.
[46] MEERITS S, THOMAS D, NOZICK V, et al. FusionMLS: highly dynamic 3D reconstruction with consumer-grade RGB-D cameras[J]. Computational Visual Media, 2018, 4: 287-303.
[47] ZHANG J, MANIATIS C, HORNA L, et al. Dynamic 3D reconstruction improvement via intensity video guided 4D fusion[J]. Journal of Visual Communication and Image Representation, 2018, 55: 540-547.
[48] FAN H, QI L, DONG J, et al. Dynamic 3D surface reconstruction using a hand-held camera[C]//Proceedings of the 44th Annual Conference of the IEEE Industrial Electronics Society, Omni Shoreham, Oct 21-23, 2018. Piscataway: IEEE, 2018: 3244-3249.
[49] SIMON T, VALMADRE J, MATTHEWS I, et al. Kronecker-Markov prior for dynamic 3D reconstruction[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 39(11): 2201-2214.
[50] 李明阳, 陈伟, 王珊珊, 等. 视觉深度学习的三维重建方法综述[J]. 计算机科学与探索, 2023, 17(2): 279-302.
LI M Y, CHEN W, WANG S S, et al. Survey on 3D reconstruction methods based on visual deep learning[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(2): 279-302.
[51] 杨丽婷, 刘孝良, 储修祥, 等. 基于MultiResHNet网络的结构光三维重建技术[J]. 激光与光电子学进展, 2023, 60(20): 149-158.
YANG L T, LIU X L, CHU X X, et al. Structured light three-dimensional reconstruction technology based on MultiResHNet[J]. Laser & Optoelectronics Progress, 2023, 60(20): 149-158.
[52] XU K, HUANG H, SHI Y, et al. Autoscanning for coupled scene reconstruction and proactive object analysis[J]. ACM Transactions on Graphics, 2015, 34(6): 1-14.
[53] ZHANG Y, XU W, TONG Y, et al. Online structure analysis for real-time indoor scene reconstruction[J]. ACM Transactions on Graphics, 2015, 34(5): 1-13.
[54] Lin C H, Wang C, Lucey S. SDF-SRN: learning signed distance 3D object reconstruction from static images[C]//Advances in Neural Information Processing Systems 33, 2020: 11453-11464.
[55] SCH?PS T, SATTLER T, H?NE C, et al. Large-scale outdoor 3D reconstruction on a mobile device[J]. Computer Vision and Image Understanding, 2017, 157: 151-166.
[56] STUTZ D, GEIGER A. Learning 3D shape completion from laser scan data with weak supervision[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Piscataway: IEEE, 2018: 1955-1964.
[57] INGALE A K. Real-time 3D reconstruction techniques applied in dynamic scenes: a systematic literature review[J]. Computer Science Review, 2021, 39: 100338.
[58] MUSTAFA A, VOLINO M, KIM H, et al. Temporally coherent general dynamic scene reconstruction[J]. International Journal of Computer Vision, 2021, 129(1): 123-141.
[59] LI J, GAO W, WU Y, et al. High-quality indoor scene 3D reconstruction with RGB-D cameras: a brief review[J]. Computational Visual Media, 2022, 8(3): 369-393.
[60] FAHIM G, AMIN K, ZARIF S. Single-view 3D reconstruction: a survey of deep learning methods[J]. Computers & Graphics, 2021, 94: 164-190.
[61] 常丽, 杨志超, 郭雨梅, 等. 改进枝切法在动态三维重建中的应用[J]. 电子测量技术, 2021, 44(9): 22-25.
CHANG L, YANG Z C, GUO Y M, et al. Application of improved branch-cut algorithm in dynamic 3D reconstruction[J]. Electronic Measurement Technology, 2021, 44(9): 22-25.
[62] SUWAJANAKORN S, KEMELMACHER-SHLIZERMAN I, SEITZ S M. Total moving face reconstruction[C]//Proceedings of the 13th European Conference on Computer Vision, Zurich, Sep 6-12, 2014. Cham: Springer, 2014: 796-812.
[63] SALZMANN M, URTASUN R, FUA P. Local deformation models for monocular 3D shape recovery[C]//Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, Jun 24-26, 2008. Piscataway: IEEE, 2008: 1-8.
[64] BOGO F, BLACK M J, LOPER M, et al. Detailed full-body reconstructions of moving people from monocular RGB-D sequences[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision, Santiago, Dec 7-13, 2015. Piscataway: IEEE, 2015.
[65] HUANG C H, ALLAIN B, FRANCO J S, et al. Volumetric 3D tracking by detection[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, Jun 26-Jul 1, 2016. Piscataway: IEEE, 2016: 3862-3870.
[66] LI H, ADAMS B, GUIBAS L J, et al. Robust single-view geometry and motion reconstruction[J]. ACM Transactions on Graphics, 2009, 28(5): 1-10.
[67] FRAGKIADAKI K, SALAS M, ARBELAEZ P, et al. Grouping-based low-rank trajectory completion and 3D reconstruction[C]//Advances in Neural Information Processing Systems 27, 2014.
[68] RUSSELL C, YU R, AGAPITO L. Video pop-up: monocular 3D reconstruction of dynamic scenes[C]//Proceedings of the 13th European Conference on Computer Vision, Zurich, Sep 6-12, 2014. Cham: Springer, 2014: 583-598.
[69] RANFTL R, VINEET V, CHEN Q, et al. Dense monocular depth estimation in complex dynamic scenes[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 26, 2016. Piscataway: IEEE, 2016: 4058-4066.
[70] DAI Y, DENG H, HE M. Dense non-rigid structure-from-motion made easy—a spatial-temporal smoothness based solution[C]//Proceedings of the 2017 IEEE International Conference on Image Processing, Beijing, Sep 17-20, 2017. Piscataway: IEEE, 2017: 4532-4536.
[71] GARG R, ROUSSOS A, AGAPITO L. Dense variational reconstruction of non-rigid surfaces from monocular video[C]//Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition, Portland, Jun 23-27, 2013. Piscataway: IEEE, 2013: 1272-1279.
[72] DIBRA E, JAINA H, OZTIRELI C, et al. Human shape from silhouettes using generative HKS descriptors and cross-modal neural networks[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Hawaii, Jul 21-26, 2017. Piscataway: IEEE, 2017: 4826-4836.
[73] ALLDIECK T, MAGNOR M, XU W, et al. Video based reconstruction of 3D people models[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Piscataway: IEEE, 2018: 8387-8397.
[74] CAI H, FENG W, FENG X, et al. Neural surface reconstruction of dynamic scenes with monocular RGB-D camera[C]//Advances in Neural Information Processing Systems 35, 2022: 967-981.
[75] CAO M, ZHENG L, LIU X. Single view 3D reconstruction based on improved RGB-D image[J]. IEEE Sensors Journal, 2020, 20(20): 12049-12056.
[76] GAO W, TEDRAKE R. Surfelwarp: efficient non-volumetric single view dynamic reconstruction[J]. arXiv:1904.13073, 2019.
[77] YU T, ZHENG Z, ZHONG Y, et al. SimulCap: single-view human performance capture with cloth simulation[C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 5504-5514.
[78] THOMAS D, SUGIMOTO A. A flexible scene representation for 3D reconstruction using an RGB-D camera[C]//Proceedings of the 2013 IEEE International Conference on Computer Vision, Sydney, Dec 1-8, 2013. Washington: IEEE Computer Society, 2013: 2800-2807.
[79] LU Y, ZHANG Z, WEN H, et al. RGB-D SLAM using scene flow in dynamic environments[C]//Proceedings of the 2022 28th International Conference on Mechatronics and Machine Vision in Practice, Nanjing, Nov 16-18, 2022. Piscataway： IEEE, 2022: 1-6.
[80] TRETSCHK E, TEWARI A, GOLYANIK V, et al. Non-rigid neural radiance fields: reconstruction and novel view synthesis of a dynamic scene from monocular video[C]//Proceedings of the 2021 IEEE International Conference on Computer Vision, Montreal, Oct 10-17, 2021. Piscataway: IEEE, 2021: 12959-12970.
[81] LI C, GUO X. Topology-change-aware volumetric fusion for dynamic scene reconstruction[C]//Proceedings of the 16th European Conference on Computer Vision, Glasgow, Aug 23-28, 2020. Cham: Springer, 2020: 258-274.
[82] 李玉梅. 基于RGB-D图像序列的动态三维重建[D]. 上海: 上海大学, 2020.
LI Y M. Dynamic 3D reconstruction basedon RGB-D image sequence[D]. Shanghai: Shanghai University, 2020.
[83] 周驰. 动态场景下基于自监督单目深度估计的稠密SLAM三维重建[D]. 长春: 吉林大学, 2022.
ZHOU C. Dense SLAM 3D reconstruction based on self-supervised monocular depth estimation in dynamic scenes [D]. Changchun: Jilin University, 2022.
[84] DAI A, NIENER M, ZOLLHFER M, et al. BundleFusion: real-time globally consistent 3D reconstruction using on-the-fly surface reintegration[J]. ACM Transactions on Graphics, 2017, 36(4): 1-18.
[85] OLEYNIKOVA H, TAYLOR Z, FEHR M, et al. Voxblox: incremental 3D euclidean signed distance fields for on-board MAV planning[C]//Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems, Vancouver, Sep 24-28, 2017. Piscataway: IEEE, 2017: 1366-1373.
[86] KELLER M, LEFLOCH D, LAMBERS M, et al. Real-time 3D reconstruction in dynamic scenes using point-based fusion[C]//Proceedings of the 2013 International Conference on 3D Vision, Seattle, Jun 29-Jul 1, 2013. Piscataway: IEEE, 2013: 1-8.
[87] JAIMEZ M, KERL C, GONZALEZ-JIMENEZ J, et al. Fast odometry and scene flow from RGB-D cameras based on geometric clustering[C]//Proceedings of the 2017 IEEE International Conference on Robotics and Automation, Singapore, May 29-Jun 3, 2017. Piscataway: IEEE, 2017: 3992-3999.
[88] GUO K, XU F, YU T, et al. Real-time geometry, albedo, and motion reconstruction using a single RGB-D camera[J]. ACM Transactions on Graphics, 2017, 36(4): 1-13.
[89] ZHANG T, ZHANG H, LI Y, et al. FlowFusion: dynamic dense RGB-D slam based on optical flow[C]//Proceedings of the 2020 IEEE International Conference on Robotics and Automation, Paris, May 31-Aug 31, 2020. Piscataway: IEEE, 2020: 7322-7328.
[90] SCONA R, JAIMEZ M, PETILLOT Y R, et al. StaticFusion: background reconstruction for dense RGB-D slam in dynamic environments[C]//Proceedings of the 2018 IEEE International Conference on Robotics and Automation, Brisbane, May 21-25, 2018. Piscataway: IEEE, 2018: 3849-3856.
[91] RUNZ M, BUFFIER M, AGAPITO L. MaskFusion: real-time recognition, tracking and reconstruction of multiple moving objects[C]//Proceedings of the 2018 IEEE International Symposium on Mixed and Augmented Reality, Munich, Oct 16-20, 2018. Piscataway: IEEE, 2018: 10-20.
[92] SLAVCHEVA M, BAUST M, ILIC S. Variational level set evolution for non-rigid 3D reconstruction from a single depth camera[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 43(8): 2838-2850.
[93] RUNZ M, AGAPITO L. Co-Fusion: real-time segmentation, tracking and fusion of multiple objects[C]//Proceedings of the 2017 IEEE International Conference on Robotics and Automation, Singapore, May 29-Jun 3, 2017. Piscataway: IEEE, 2017: 4471-4478.
[94] ZHANG H, XU F. MixedFusion: real-time reconstruction of an indoor scene with dynamic objects[J]. IEEE Transactions on Visualization and Computer Graphics, 2017, 24(12): 3137-3146.
[95] LIU Y, PENG X, ZHOU W, et al. Template-based 3D reconstruction of non-rigid deformable object from monocular video[J]. 3D Research, 2018, 9: 1-12.
[96] YU R, RUSSELL C, CAMPBELL N D F, et al. Direct, dense, and deformable: template-based non-rigid 3D reconstruction from RGB video[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision, Santiago, Dec 7-13, 2015. Piscataway: IEEE, 2015: 918-926.
[97] PENG S D, DONG J T, WANG Q Q, et al. Animatable neural radiance fields for modeling dynamic human bodies[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, Montreal, Oct 10-17, 2021. Piscataway: IEEE, 2021: 14314-14323.
[98] XU W, CHATTERJEE A, ZOLLH?FER M, et al. Monoperfcap: human performance capture from monocular video[J]. ACM Transactions on Graphics, 2018, 37(2): 1-15.
[99] QU C, GAO H, EKENEL H K. Rotation update on manifold in probabilistic NRSFM for robust 3D face modeling[J]. EURASIP Journal on Image and Video Processing, 2015(1): 1-12.
[100] ZHU Y, HUANG D, DE LA TORRE F, et al. Complex non-rigid motion 3D reconstruction by union of subspaces[C]//Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, Jun 23-28, 2014. Piscataway: IEEE, 2014: 1542-1549.
[101] KUMAR S, DAI Y, LI H. Spatio-temporal union of subspaces for multi-body non-rigid structure-from-motion[J]. Pattern Recognition, 2017, 71: 428-443.
[102] SLAVCHEVA M, BAUST M, ILIC S. SobolevFusion: 3D reconstruction of scenes undergoing free non-rigid motion[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Piscataway: IEEE, 2018: 2646-2655.
[103] ZOLLHOFER M, STOTKO P, GORLITZ A, et al. State of the art on 3D reconstruction with RGB-D cameas[J]. Computer Graphics Forum, 2018, 37(2): 625-652.
[104] HABERMANN M, XU W, ZOLLHOEFER M, et al. Livecap: real-time human performance capture from monocular video[J]. ACM Transactions on Graphics, 2019, 38(2): 1-17.
[105] XIANG D, JOO H, SHEIKH Y. Monocular total capture: posing face, body, and hands in the wild[C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 10965-10974.
[106] KOCABAS M, ATHANASIOU N, BLACK M J. Vibe: video inference for human body pose and shape estimation[C]//Proceedings of the 2020 IEEE Conference on Computer Vision and Pattern Recognition, Seattle, Jun 14-19, 2020. Piscataway: IEEE, 2020: 5253-5263.
[107] LI H, YU J, YE Y, et al. Realtime facial animation with on-the-fly correctives[J]. ACM Transactions on Graphics, 2013, 32(4): 1-10.
[108] TKACH A, PAULY M, TAGLIASACCHI A. Sphere-meshes for real-time hand modeling and tracking[J]. ACM Transactions on Graphics, 2016, 35(6): 1-11.
[109] PONS-MOLL G, BAAK A, HELTEN T, et al. Multisensor-fusion for 3D full-body human motion capture[C]//Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, Jun 13-18, 2010. Piscataway: IEEE, 2010: 663-670.
[110] ZHANG J, LI X, WAN Z, et al. FDNeRF: few-shot dynamic neural radiance fields for face reconstruction and expression editing[C]//Proceedings of the SIGGRAPH Asia 2022 Conference Papers, Vancouver, Aug 8-11, 2022. New York: ACM, 2022.
[111] GANPULE S, DAPHALAPURKAR N P, RAMESH K T, et al. A three-dimensional computational human head model that captures live human brain dynamics[J]. Journal of Neurotrauma, 2017, 34(13): 2154-2166.
[112] WANG K, ZHANG G, YANG J, et al. Dynamic human body reconstruction and motion tracking with low-cost depth cameras[J]. The Visual Computer, 2021, 37: 603-618.
[113] SIDHU V, TRETSCHK E, GOLYANIK V, et al. Neural dense non-rigid structure from motion with latent space constraints[C]//Proceedings of the 16th European Conference on Computer Vision, Glasgow, Aug 23-28, 2020. Cham: Springer, 2020: 204-222.
[114] GOLYANIK V. Robust methods for dense monocular non-rigid 3D reconstruction and alignment of point clouds[M]. Berlin: Springer Nature, 2020.
[115] SLAVCHEVA M, BAUST M, CREMERS D, et al. KillingFusion: non-rigid 3D reconstruction without correspondences[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Piscataway: IEEE, 2017: 5474-5483.
[116] KOZLOV C, SLAVCHEVA M, ILIC S. Patch-based non-rigid 3D reconstruction from a single depth stream[C]//Proceedings of the 2018 International Conference on 3D Vision, Verona, Sep 5-8, 2018. Piscataway: IEEE, 2018: 42-51.
[117] AGUDO A. Segmentation and 3D reconstruction of non-rigid shape from RGB video[C]//Proceedings of the 2020 IEEE International Conference on Image Processing, Abu Dhabi, Oct 25-28, 2020. Piscataway: IEEE, 2020: 2845-2849.
[118] MALTI A, HARTLEY R, BARTOLI A, et al. Monocular template-based 3D reconstruction of extensible surfaces with local linear elasticity[C]//Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition, Portland, Jun 23-27, 2013. Piscataway: IEEE, 2013: 1522-1529.
[119] ZOLLHOFER M, NIE?NER M, IZADI S, et al. Real-time non-rigid reconstruction using an RGB-D camera[J]. ACM Transactions on Graphics, 2014, 33(4): 1-12.
[120] 刘洋. 基于单目视频序列的非刚性动态目标三维重建算法研究[D]. 北京: 中国科学院大学(中国科学院国家空间科学中心), 2018.
LIU Y. Research on 3D non-rigid reconstruction of dynamic and deforming object using monocular video sequence[D]. Beijing: University of Chinese Academy of Sciences (National Space Science Center), 2018.
[121] BREGLER C, HERTZMANN A, BIERMANN H. Recovering non-rigid 3D shape from image streams[C]//Proceedings of the 2020 IEEE Conference on Computer Vision and Pattern Recognition, Hilton Head, Jun 15-15, 2020. Piscataway: IEEE, 2000: 690-696.
[122] AKHTER I, SHEIKH Y, KHAN S, et al. Trajectory space: a dual representation for nonrigid structure from motion[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(7): 1442-1456.
[123] DAI Y, LI H, HE M. A simple prior-free method for non-rigid structure-from-motion factorization[J]. International Journal of Computer Vision, 2014, 107: 101-122.
[124] WANG Y P, SUN Z L, LAM K M. An effective approach for NRSFM of small-size image sequences[J]. PLoS One, 2015, 10(7): e0132370.
[125] 杨苏. 基于NRSFM的动态场景三维重建方法研究[D]. 西安: 陕西科技大学, 2020.
YANG S. Research on 3D reconstruction of dynamic scene based on NRSFM[D]. Xi’an: Shaanxi University of Science & Technology, 2020.
[126] TAYLOR J, BORDEAUX L, CASHMAN T, et al. Efficient and precise interactive hand tracking through joint, continuous optimization of pose and correspondences[J]. ACM Transactions on Graphics, 2016, 35(4): 1-12.
[127] DOU M, KHAMIS S, DEGTYAREV Y, et al. Fusion4D: real-time performance capture of challenging scenes[J]. ACM Transactions on Graphics, 2016, 35(4): 1-13.
[128] LU F, ZHOU B, ZHANG Y, et al. Real-time 3D scene reconstruction with dynamically moving object using a single depth camera[J]. The Visual Computer, 2018, 34: 753-763.
[129] LI Y, ZHANG T, NAKAMURA Y, et al. SplitFusion: simultaneous tracking and mapping for non-rigid scenes[C]//Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems, Las Vegas, Oct 24-Jan 24, 2020. Piscataway: IEEE, 2020: 5128-5134.
[130] 李翔宇, 张雪芹. ORBTSDF-SCNet: 一种动态场景在线三维重建方法[J]. 华东理工大学学报(自然科学版), 2023, 49(2): 284-294.
LI X Y, ZHANG X Q. QORBTSDF-SCNet: an online 3D reconstruction method for dynamic scene[J]. Journal of East China University of Science and Technology (Natural Science Edition), 2023, 49(2): 284-294.
[131] PAN Z, HOU J, YU L. Optimization RGB-D 3-D reconstruction algorithm based on dynamic SLAM[J]. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 1-13.
[132] ALJAZ B, MICHAEL Z, CHRISTIAN T, et al. Deep-Deform: learning non-rigid RGB-D reconstruction with semi-supervised data[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 14-19, 2020. Piscataway: IEEE, 2020: 7002-7012.
[133] STURM J, ENGELHARD N, ENDRES F, et al. A benchmark for the evaluation of RGB-D SLAM systems[C]//Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, Vilamoura, Oct 7-12, 2012. Piscataway: IEEE Press, 2012: 573-580.
[134] SILBERMAN N, FERGUS R. Indoor scene segmentation using a structured light sensor[C]//Proceedings of the 2011 IEEE International Conference on Computer Vision Workshops, Barcelona, Nov 6-13, 2011. Piscataway: IEEE, 2011.
[135] ARMENI I, SENER O, ZAMIR A R, et al. 3D semantic parsing of large-scale indoor spaces[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 26-Jul 1, 2016. Piscataway: IEEE, 2016: 1534-1543.
[136] GLOCKER B, IZADI S, SHOTTON J, et al. Real-time RGB-D camera relocalization[C]//Proceedings of the 2013 IEEE International Symposium on Mixed and Augmented Reality, Adelaide, Oct 1-4, 2013. Piscataway: IEEE, 2013: 173-179.
[137] PHAM Q H, UY M A, HUA B S, et al. LCD: learned cross-domain descriptors for 2D-3D matching[C]//Proceedings of the 2020 AAAI Conference on Artificial Intelligence, New York, Feb 7-12, 2020. Menlo Park: AAAI, 2020: 11856-11864.
[138] SCHOPS T, SATTLER T, POLLEFEYS M. BAD SLAM: bundle adjusted direct RGB-D slam[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 134-144.
[139] HANDA A, WHELAN T, MCDONLA J, et al. A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM[C]//Proceedings of the 2014 IEEE International Conference on Robotics and Automation, Hong Kong, China, May 31-Jun 7, 2014. Piscataway: IEEE, 2014: 1524-1531.
[140] DANIEL V, ILYA B, WOJCIECH M, et al. Articulated mesh animation from multi-view silhouettes[J]. ACM Transactions on Graphics, 2008, 27(3): 1-9.
[141] EBNER T, FELDMANN I, RENAULT S, et al. Multi-view reconstruction of dynamic real-world objects and their integration in augmented and virtual reality applications[J]. Journal of the Society for Information Display, 2017, 25(3): 151-157.
[142] WONG B, SPETSAKIS M. Scene reconstruction and robot navigation using dynamic fields[J]. Autonomous Robots, 2000, 8: 71-86.
[143] MALIK A, LHACHEMI H, PLOENNIGS J, et al. An application of 3D model reconstruction and augmented reality for real-time monitoring of additive manufacturing[J]. Procedia Cirp, 2019, 81: 346-351.
[144] QIN C, SCHLEMPER J, CABALLERO J, et al. Convolutional recurrent neural networks for dynamic MR image reconstruction[J]. IEEE Transactions on Medical Imaging, 2018, 38(1): 280-290.
[145] SCHLEMPER J, CABALLERO J, HAJNAL J V, et al. A deep cascade of convolutional neural networks for MR image reconstruction[C]//Proceedings of the 25th International Conference on Information Processing in Medical Imaging, Boone, Jun 25-30, 2017. Berlin: Springer International Publishing, 2017: 647-658.
[146] MATEO C M, CORRALES J A, MEZOUAR Y. Hierarchical, dense and dynamic 3D reconstruction based on VDB data structure for robotic manipulation Tasks[J]. Frontiers in Robotics and AI, 2021, 7: 600387.
[147] ORTS-ESCOLANO S, RHEMANN C, FANELLO S, et al. Holoportation: virtual 3D teleportation in real-time[C]//Proceedings of the 29th Annual Symposium on User Interface Software and Technology, Tokyo, Oct 16-19, 2016. New York: ACM, 2016: 741-754.
[148] 吕东岳, 黄志蓓, 陶冠宏, 等. 使用简易深度成像设备的高尔夫挥杆动态贝叶斯网络三维重建[J]. 电子与信息学报, 2015, 37(9): 2076-2081.
LV D Y, HUANG Z P, TAO G H, et al. Dynamic Bayesian network model based golf swing 3D reconstruction using simple depth imaging device[J]. Journal of Electronics & Information Technology, 2015, 37(9): 2076-2081.