Review of Research on Fatigue Driving Detection Based on Driver Facial Features

doi:10.3778/j.issn.1673-9418.2208041

Abstract

Abstract: Fatigue driving is one of the main factors that threaten the safety of drivers and traffic. Efficient and accurate fatigue driving detection method can effectively ensure the safety of drivers and their surrounding traffic, maintain traffic order, and reduce property losses and casualties. The fatigue driving detection method based on the driver’s physiological characteristics and vehicle driving information has many limitations, such as being unfriendly to the driver and having numerous influencing factors. Therefore, the fatigue driving detection method based on the driver’s facial features has become a research hotspot. Firstly, the facial feature performance of fatigue driving is described, and advantages, disadvantages and application scenarios of common public datasets in the field of fatigue driving are summarized. Secondly, the advantages and disadvantages of common face detection algorithms in the field of fatigue driving detection are analyzed and studied by using open datasets and comparative experiments. Then, this paper generalizes the process of detection methods based on driver facial features, whose methods and technologies used in the key steps are reviewed. Furthermore, fatigue discriminant parameters and methods of fatigue driving results prediction are summarized. Finally, this paper ends with a discussion of current challenges of fatigue driving detection methods based on driver facial features and looks forward to the future research.

Key words: driver facial features, fatigue driving test, fatigue discriminant parameters, feature extraction, face detection

摘要： 疲劳驾驶是威胁驾驶员人身安全以及道路交通安全的主要因素之一。高效精准的疲劳驾驶检测方法可以有效地保障驾驶员及其周围交通安全，维护交通秩序，减少财产损失和人员伤亡。由于基于驾驶员生理特征和基于车辆行驶信息的疲劳驾驶检测方法具有对驾驶员不友好、影响因素较多等局限性，使得基于驾驶员面部特征的疲劳驾驶检测方法成为研究热点。首先简述了疲劳驾驶面部特征表现，总结了疲劳驾驶领域常用公开数据集的优缺点和应用场景；其次使用公开数据集，通过对比实验，分析研究了疲劳驾驶检测领域常用人脸检测算法的优势和不足；随后给出了基于驾驶员面部特征的疲劳驾驶检测方法流程，总结分析了流程中关键步骤所使用的方法和技术；另外归纳整理了疲劳驾驶领域常用的疲劳判别参数和疲劳驾驶结果预测方法；最后对全文进行总结，给出了基于驾驶员面部特征的疲劳驾驶检测方法目前所面临的挑战，并对未来研究进行了展望。

关键词: 驾驶员面部特征, 疲劳驾驶检测, 疲劳判别参数, 特征提取, 人脸检测

YANG Yanyan, LI Leixiao, LIN Hao. Review of Research on Fatigue Driving Detection Based on Driver Facial Features[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(6): 1249-1267.

杨艳艳, 李雷孝, 林浩. 提取驾驶员面部特征的疲劳驾驶检测研究综述[J]. 计算机科学与探索, 2023, 17(6): 1249-1267.

References

[1] AMODIO A, ERMIDORO M, MAGGI D, et al. Automatic detection of driver impairment based on pupillary light reflex[J]. IEEE Transactions on Intelligent Transportation Systems, 2019, 20(8): 3038-3048.
[2] ZHENG W L, GAO K P, LI G, et al. Vigilance estimation using a wearable EOG device in real driving environment[J]. IEEE Transactions on Intelligent Transportation Systems, 2020, 21(1): 170-184.
[3] ZHU M, CHEN J, LI H, et al. Vehicle driver drowsiness detection method using wearable EEG based on convolu-tion neural network[J]. Neural Computing & Applications, 2021, 33(20): 13965-13980.
[4] AREFNEZHAD S, SAMIEE S, EICHBERGER A, et al. Applying deep neural networks for multi-level classification of driver drowsiness using vehicle-based measures[J]. Expert Systems with Applications, 2020, 162(1): 113778.
[5] JEON Y, KIM B, BAEK Y. Ensemble CNN to detect drowsy driving with in-vehicle sensor data[J]. Sensors, 2021, 21(7): 2372.
[6] MUHAMMAD R, KHAN H U, AWAN S M, et al. A survey on state-of-the-art drowsiness detection techniques[J]. IEEE Access, 2019, 7: 61904-61919.
[7] KAMRAN M A, MANNAN M M N, JEONG M Y. Drow-siness, fatigue and poor sleep’s causes and detection: a com-prehensive study[J]. IEEE Access, 2019, 7: 167172-167186.
[8] 张佐营, 叶桂荀. 驾驶疲劳监测技术研究综述[J]. 汽车科技, 2022(1): 8-14.
ZHANG Z Y, YE G X. Review on driving fatigue detection[J]. Automobile Science & Technology, 2022(1): 8-14.
[9] ARAKAWA T. Trends and future prospects of the drowsi-ness detection and estimation technology[J]. Sensors, 2021, 21(23): 7921.
[10] 张瑞, 朱天军, 邹志亮, 等. 驾驶员疲劳驾驶检测方法研究综述[J]. 计算机工程与应用, 2022, 58(21): 53-66.
ZHANG R, ZHU T J, ZOU Z L, et al. Review of research on driver fatigue driving detection methods[J]. Computer Engineering and Applications, 2022, 58(21): 53-66.
[11] NEMCOVA A, SVOZILOVA V, BUCSUHAZY K, et al. Multimodal features for detection of driver stress and fati-gue: review[J]. IEEE Transactions on Intelligent Transporta-tion Systems, 2021, 22(6): 3214-3233.
[12] LI K, GONG Y, REN Z. A fatigue driving detection algori-thm based on facial multi-feature fusion[J]. IEEE Access, 2020, 8: 101244-101259.
[13] 史瑞鹏, 钱屹, 蒋丹妮. 一种基于卷积神经网络的疲劳驾驶检测方法[J]. 计算机应用研究, 2020, 37(11): 3481-3486.
SHI R P, QIAN Y, JIANG D N. Fatigue driving detection method based on CNN[J]. Application Research of Compu-ters, 2020, 37(11): 3481-3486.
[14] BAKKER B, ZABLOCKI B, BAKER A, et al. A multi-stage, multi-feature machine learning approach to detect driver sleepiness in naturalistic road driving conditions[J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 23(5): 1-10.
[15] 冯文文, 曹银杰, 李晓琳, 等. 基于改进的深度卷积神经网络的人脸疲劳检测[J]. 科学技术与工程, 2020, 20(14): 5680-5687.
FENG W W, CAO Y J, LI X L, et al. Face fatigue detection based on improved deep convolutional neural network[J]. Science Technology and Engineering, 2020, 20(14): 5680-5687.
[16] ZHANG C, WU X P, ZHENG X, et al. Driver drowsiness detection using multi-channel second order blind identifica-tions[J]. IEEE Access, 2019, 7: 11829-11843.
[17] 潘志庚, 刘荣飞, 张明敏. 基于模糊综合评价的疲劳驾驶检测算法研究[J]. 软件学报, 2019, 30(10): 2954-2963.
PAN Z G, LIU R F, ZHANG M M. Research on fatigue driving detection algorithm based on fuzzy comprehensive evaluation[J]. Journal of Software, 2019, 30(10): 2954-2963.
[18] JI Y Y, WANG S G, ZHAO Y, et al. Fatigue state detection based on multi-index fusion and state recognition network[J]. IEEE Access, 2019, 7: 64136-64147.
[19] 董超俊, 林庚华, 吴承鑫, 等. 基于卷积专家神经网络的疲劳驾驶检测[J]. 计算机工程与设计, 2020, 41(10): 2812-2817.
DONG C J, LIN G H, WU C X, et al. Fatigue driving detection based on convolution expert neural network[J]. Computer Engineering and Design, 2020, 41(10): 2812-2817.
[20] 徐莲, 任小洪, 陈闰雪. 基于眼睛状态识别的疲劳驾驶检测[J]. 科学技术与工程, 2020, 20(20): 8292-8299.
XU L, REN X H, CHEN R X. Fatigue driving detection based on eye state recognition[J]. Science Technology and Engineering, 2020, 20(20): 8292-8299.
[21] LING Y C, LUO R, DONG X, et al. Driver eye location and state estimation based on a robust model and data augmen-tation[J]. IEEE Access, 2021, 9: 67219-67231.
[22] 陆荣秀, 张笔豪, 莫振龙. 基于脸部特征和头部姿态的疲劳检测方法[J]. 系统仿真学报, 2022, 34(10): 2279-2292.
LU R X, ZHANG B H, MO Z L. Fatigue detection method based on facial features and head posture[J]. Journal of System Simulation, 2022, 34(10): 2279-2292.
[23] 胡习之, 黄冰瑜. 基于面部特征分析的疲劳驾驶检测方法[J]. 科学技术与工程, 2021, 21(4): 1629-1636.
HU X Z, HUANG B Y. Fatigue driving detection system based on face feature analysis[J]. Science Technology and Engineering, 2021, 21(4): 1629-1636.
[24] 娄平, 杨欣, 胡辑伟, 等. 基于边缘计算的疲劳驾驶检测方法[J]. 计算机工程, 2021, 47(7): 13-20.
LOU P, YANG X, HU J W, et al. Fatigue driving detection method based on edge computing[J]. Computer Enginee-ring, 2021, 47(7): 13-20.
[25] 潘剑凯, 柳政卿, 王秋成. 基于眼部自商图-梯度图共生矩阵的疲劳驾驶检测[J]. 中国图象图形学报, 2021, 26(1): 154-164.
PAN J K, LIU Z Q, WANG Q C. Fatigue driving detection based on ocular self-quotient image and gradient image co-occurrence matrix[J]. Journal of Image and Graphics, 2021, 26(1): 154-164.
[26] 庄员, 戚湧. 伪3D卷积神经网络与注意力机制结合的疲劳驾驶检测[J]. 中国图象图形学报, 2021, 26(1): 143-153.
ZHUANG Y, QI Y. Driving fatigue detection based on pseudo 3D convolutional neural network and attention mec-hanisms[J]. Journal of Image and Graphics, 2021, 26(1): 143-153.
[27] YANG H, LIU L, MIN W D, et al. Driver yawning detection based on subtle facial action recognition[J]. IEEE Transac-tions on Multimedia, 2021, 23: 572-583.
[28] 敖邦乾, 杨莎, 令狐金卿, 等. 基于级联神经网络疲劳驾驶检测系统设计[J]. 系统仿真学报, 2022, 34(2): 323-333.
AO B Q, YANG S, LINGHU J Q, et al. Design of fatigue driving detection system based on cascaded neural network[J]. Journal of System Simulation, 2022, 34(2): 323-333.
[29] HUANG R, WANG Y, LI Z J, et al. RF-DCM: multi-granu-larity deep convolutional model based on feature recalibra-tion and fusion for driver fatigue detection[J]. IEEE Tran-sactions on Intelligent Transportation Systems, 2022, 23(1): 630-640.
[30] SONG F Y, TAN X Y, LIU X, et al. Eyes closeness detec-tion from still images with multi-scale histograms of princi-pal oriented gradients[J]. Pattern Recognition, 2014, 47(9): 2825-2838.
[31] FUSEK R. Pupil localization using geodesic distance[C]//LNCS 11241: Proceedings of the International Symposium on Visual Computing, Las Vegas, Nov 19-21, 2018: 433-444.
[32] ABTAHI S, OMIDYEGANEH M, SHIRMOHAMMADI S, et al. YawDD: a yawning detection dataset[C]//Proceedings of the 5th ACM Multimedia Systems Conference, Singa-pore, Mar 19-21, 2014. New York: ACM, 2014: 24-28.
[33] WENG C H, LAI Y H, LAI S H. Driver drowsiness detec-tion via a hierarchical temporal deep belief network[C]//LNCS 10118: Proceedings of the 2016 International Work-shops on Computer Vision, Taipei, China, Nov 20-24, 2016. Cham: Springer, 2016: 117-133.
[34] GHODDOOSIAN R, GALIB M, ATHITSOS V. A realistic dataset and baseline temporal model for early drowsiness detection[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Long Beach, Jun 15-20, 2019. Piscataway: IEEE, 2019: 178-187.
[35] BRADSKI G. The OpenCV library[J]. Doctor Dobbs Journal, 2000: 25.
[36] VIOLA P, JONES M J. Robust real-time face detection[J]. International Journal of Computer Vision, 2004, 57(2): 137-154.
[37] KING D E. Dlib-ml: a machine learning toolkit[J]. Journal of Machine Learning Research, 2009, 10(3): 1755-1758.
[38] DALAL N, TRIGGS B. Histograms of oriented gradients for human detection[C]//Proceedings of the 2005 IEEE Com-puter Society Conference on Computer Vision and Pattern Recognition, San Diego, Jun 20-25, 2005. Washington: IEEE Computer Society, 2005: 886-893.
[39] LIU W, ANGUELOV D, ERHAND, et al. SSD: single shot multibox detector[C]//LNCS 9905: Proceedings of the 14th European Conference on Computer Vision, Amsterdam, Oct 11-14, 2016. Cham: Springer, 2016: 21-37.
[40] ZHANG K, ZHANG Z, LI Z, et al. Joint face detection and alignment using multitask cascaded convolutional networks[J]. IEEE Signal Processing Letters, 2016, 23(10): 1499-1503.
[41] SOUKUPOVA T, CECH J. Real-time eye blink detection using facial landmarks[C]//Proceedings of the 21st Com-puter Vision Winter Workshop, Rimske Toplice, Feb 3-5, 2016.
[42] RAMOS A L A, ERANDIO J C, MANGILAVA D H T, et al. Driver drowsiness detection based on eye movement and yawning using facial landmark analysis[J]. Internatio-nal Journal of Simulation: Systems, Science and Techno-logy, 2019, 20: 371-378.
[43] KONG X J, XIA F, LI J X, et al. A shared bus profiling scheme for smart cities based on heterogeneous mobile crowd-sourced data[J]. IEEE Transactions on Industrial Informa-tics, 2020, 16(2): 1436-1444.
[44] NING Z L, HUANG J, WANG X J, et al. Mobile edge com-puting-enabled internet of vehicles: toward energy-efficient scheduling[J]. IEEE Network, 2019, 33(5): 198-205.
[45] FENG Y, LI X L, GONG Y B, et al. A real-time driving drow-siness detection algorithm with individual differences consi-deration[J]. IEEE Access, 2019, 7: 179396-179408.
[46] ABTAHI S, HARIRI B, SHIRMOHAMMADI S. Driver drowsiness monitoring based on yawning detection[C]//Proceedings of the 2011 IEEE International Instrumenta-tion and Measurement Technology Conference, Hangzhou, May 10-12, 2011. Piscataway: IEEE, 2011: 1-4.
[47] IBRAHM M M, SOROGHAN J S, PETROPOULAKIS L. Mouth covered detection for yawn[C]//Proceedings of the 2013 IEEE International Conference on Signal and Image Processing Applications, Melaka, Oct 8-10, 2013. Piscataway: IEEE, 2013: 89-94.
[48] OMIDYEGANEH M, JAVADTALAB A, SHIRMOHAMM-ADI S. Intelligent driver drowsiness detection through fusion of yawning and eye closure[C]//Proceedings of the 2011 IEEE International Conference on Virtual Environ-ments, Human-Computer Interfaces and Measurement Sys-tems, Ottawa, Sep 19-21, 2011. Piscataway: IEEE, 2011: 18-23.
[49] OMIDYEGANEH M, SHIRMOHAMMADI S, ABTAHI S, et al. Yawning detection using embedded smart cameras[J]. IEEE Transactions on Instrumentation and Measurement, 2016, 65(3): 570-582.
[50] RABMAN A, SIRSHAR M, KHAN A. Real time drowsi-ness detection using eye blink monitoring[C]//Proceedings of the 2015 National Software Engineering Conference, Rawalpindi, Dec 17-17, 2015. Piscataway: IEEE, 2015: 1-7.
[51] ZHUANG Q Y, ZHANG K H, WANG J Y, et al. Driver fatigue detection method based on eye states with pupil and iris segmentation[J]. IEEE Access, 2020, 8: 173440-173449.
[52] LU Y F, LI C L. Recognition of driver eyes?? states based on variance projections function[C]//Proceedings of the 2010 3rd International Congress on Image and Signal Proces-sing, Yantai, 2010. Piscataway: IEEE, 2010: 1919-1922.
[53] HUANG R, WANG Y, GUO L. P-FDCN based eye state analysis for fatigue detection[C]//Proceedings of the 2018 IEEE 18th International Conference on Communication Tech-nology, Chongqing, Oct 8-11, 2018. Piscataway: IEEE, 2018: 1174-1178.
[54] 李科岑, 王晓强, 林浩, 等. 深度学习中的单阶段小目标检测方法综述[J]. 计算机科学与探索, 2022, 16(1): 41-58.
LI K C, WANG X Q, LIN H, et al. Survey of one-stage small object detection methods in deep learning[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(1): 41-58.
[55] 王鑫鹏, 王晓强, 林浩, 等. 深度学习典型目标检测算法的改进综述[J]. 计算机工程与应用, 2022, 58(6): 42-57.
WANG X P, WANG X Q, LIN H, et al. Review on impro-vement of typical object detection algorithms in deep lear-ning[J]. Computer Engineering and Applications, 2022, 58(6): 42-57.
[56] FU R R, WANG H, ZHAO W B. Dynamic driver fatigue detection using hidden Markov model in real driving condi-tion[J]. Expert Systems with Applications, 2016, 63: 397-411.
[57] DU G L, LI T, LI C Q, et al. Vision-based fatigue driving recognition method integrating heart rate and facial features[J]. IEEE Transactions on Intelligent Transportation Sys-tems, 2021, 22(5): 3089-3100.