计算机科学与探索 ›› 2022, Vol. 16 ›› Issue (5): 1146-1154.DOI: 10.3778/j.issn.1673-9418.2104106
收稿日期:
2021-04-29
修回日期:
2021-08-05
出版日期:
2022-05-01
发布日期:
2022-05-19
通讯作者:
+ E-mail: cheng_weiyue@sina.cn作者简介:
程卫月(1988—),女,黑龙江哈尔滨人,硕士,讲师,主要研究方向为稀疏表示、图像处理、模式识别等。基金资助:
CHENG Weiyue1,+(), ZHANG Xueqin2, LIN Kezheng2, LI Ao2
Received:
2021-04-29
Revised:
2021-08-05
Online:
2022-05-01
Published:
2022-05-19
About author:
CHENG Weiyue, born in 1988, M.S., lecturer. Her research interests include sparse representation, image processing, pattern recognition, etc.Supported by:
摘要:
为进一步提高人脸表情识别的准确率,提出一种融合全局与局部特征的深度卷积神经网络算法(GL-DCNN)。该算法由两个改进的卷积神经网络分支组成,全局分支和局部分支,分别用于提取全局特征和局部特征,对两个分支的特征进行加权融合,使用融合后的特征进行分类。首先,提取全局特征,全局分支基于迁移学习,使用改进的VGG19网络模型进行特征提取;其次,提取局部特征,局部分支采用中心对称局部二值模式(CSLBP)算法进行第一次特征提取,得到原始图像的局部纹理信息,将其输入到浅层卷积神经网络进行第二次特征提取,使其自动提取出与表情相关的局部特征;再次,采用两个级联的全连接层对两个分支的特征进行降维,为其分配不同权重,进行加权融合;最后,采用softmax分类器进行分类。实验在CK+和JAFFE数据集上进行验证,分类精度分别达95%以上和93%以上,对比其他五种算法,该算法总体表现较好,具有较好的识别效果和良好的鲁棒性,可为人脸表情识别提供有效依据。
中图分类号:
程卫月, 张雪琴, 林克正, 李骜. 融合全局与局部特征的深度卷积神经网络算法[J]. 计算机科学与探索, 2022, 16(5): 1146-1154.
CHENG Weiyue, ZHANG Xueqin, LIN Kezheng, LI Ao. Deep Convolutional Neural Network Algorithm Fusing Global and Local Features[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(5): 1146-1154.
网络层 | 输入大小 | 参数 | 输出大小 |
---|---|---|---|
Conv-1 | 64×64×3 | 7×7×64-1-3 | 64×64×64 |
MaxPool-1 | 64×64×64 | 2×2-2-0 | 32×32×64 |
Dropout1 | 32×32×64 | — | 32×32×64 |
Conv-2 | 32×32×64 | 3×3×256-1-0 | 32×32×256 |
MaxPool-2 | 32×32×256 | 2×2-2-0 | 16×16×256 |
Dropout2 | 16×16×256 | — | 16×16×256 |
FC | 1×500 | — | 1×500 |
表1 CNN的网络配置
Table 1 Configuration of CNN
网络层 | 输入大小 | 参数 | 输出大小 |
---|---|---|---|
Conv-1 | 64×64×3 | 7×7×64-1-3 | 64×64×64 |
MaxPool-1 | 64×64×64 | 2×2-2-0 | 32×32×64 |
Dropout1 | 32×32×64 | — | 32×32×64 |
Conv-2 | 32×32×64 | 3×3×256-1-0 | 32×32×256 |
MaxPool-2 | 32×32×256 | 2×2-2-0 | 16×16×256 |
Dropout2 | 16×16×256 | — | 16×16×256 |
FC | 1×500 | — | 1×500 |
网络层 | 输入大小 | 参数 | 输出大小 |
---|---|---|---|
Conv-1 | 64×64×3 | 3×3×64-1-1 | 64×64×64 |
Conv-2 | 64×64×64 | 3×3×64-1-1 | 64×64×64 |
MaxPool-1 | 64×64×64 | 2×2-2-0 | 32×32×64 |
Dropout1 | 32×32×64 | — | 32×32×64 |
Conv-3 | 32×32×64 | 3×3×128-1-1 | 32×32×128 |
Conv-4 | 32×32×128 | 3×3×128-1-1 | 32×32×128 |
MaxPool-2 | 32×32×128 | 2×2-2-0 | 16×16×128 |
Dropout2 | 16×16×128 | — | 16×16×128 |
Conv-5 | 16×16×128 | 3×3×256-1-1 | 16×16×256 |
Conv-6 | 16×16×256 | 3×3×256-1-1 | 16×16×256 |
Conv-7 | 16×16×256 | 3×3×256-1-1 | 16×16×256 |
Conv-8 | 16×16×256 | 3×3×256-1-1 | 16×16×256 |
MaxPool-3 | 16×16×256 | 2×2-2-0 | 8×8×256 |
Dropout3 | 8×8×256 | — | 8×8×256 |
Conv-9 | 8×8×256 | 3×3×512-1-1 | 8×8×512 |
Conv-10 | 8×8×512 | 3×3×512-1-1 | 8×8×512 |
Conv-11 | 8×8×512 | 3×3×512-1-1 | 8×8×512 |
Conv-12 | 8×8×512 | 3×3×512-1-1 | 8×8×512 |
MaxPool-4 | 8×8×512 | 2×2-2-0 | 4×4×512 |
Dropout4 | 4×4×512 | — | 4×4×512 |
Conv-13 | 4×4×512 | 3×3×512-1-1 | 4×4×512 |
Conv-14 | 4×4×512 | 3×3×512-1-1 | 4×4×512 |
Conv-15 | 4×4×512 | 3×3×512-1-1 | 4×4×512 |
Conv-16 | 4×4×512 | 3×3×512-1-1 | 4×4×512 |
MaxPool-5 | 4×4×512 | 2×2-2-0 | 2×2×512 |
Dropout5 | 2×2×512 | — | 2×2×512 |
FC-1 | 2×2×512 | — | 500 |
Dropout6 | 1×500 | — | 1×500 |
表2 改进的VGG19网络配置
Table 2 Configuration of improved VGG19
网络层 | 输入大小 | 参数 | 输出大小 |
---|---|---|---|
Conv-1 | 64×64×3 | 3×3×64-1-1 | 64×64×64 |
Conv-2 | 64×64×64 | 3×3×64-1-1 | 64×64×64 |
MaxPool-1 | 64×64×64 | 2×2-2-0 | 32×32×64 |
Dropout1 | 32×32×64 | — | 32×32×64 |
Conv-3 | 32×32×64 | 3×3×128-1-1 | 32×32×128 |
Conv-4 | 32×32×128 | 3×3×128-1-1 | 32×32×128 |
MaxPool-2 | 32×32×128 | 2×2-2-0 | 16×16×128 |
Dropout2 | 16×16×128 | — | 16×16×128 |
Conv-5 | 16×16×128 | 3×3×256-1-1 | 16×16×256 |
Conv-6 | 16×16×256 | 3×3×256-1-1 | 16×16×256 |
Conv-7 | 16×16×256 | 3×3×256-1-1 | 16×16×256 |
Conv-8 | 16×16×256 | 3×3×256-1-1 | 16×16×256 |
MaxPool-3 | 16×16×256 | 2×2-2-0 | 8×8×256 |
Dropout3 | 8×8×256 | — | 8×8×256 |
Conv-9 | 8×8×256 | 3×3×512-1-1 | 8×8×512 |
Conv-10 | 8×8×512 | 3×3×512-1-1 | 8×8×512 |
Conv-11 | 8×8×512 | 3×3×512-1-1 | 8×8×512 |
Conv-12 | 8×8×512 | 3×3×512-1-1 | 8×8×512 |
MaxPool-4 | 8×8×512 | 2×2-2-0 | 4×4×512 |
Dropout4 | 4×4×512 | — | 4×4×512 |
Conv-13 | 4×4×512 | 3×3×512-1-1 | 4×4×512 |
Conv-14 | 4×4×512 | 3×3×512-1-1 | 4×4×512 |
Conv-15 | 4×4×512 | 3×3×512-1-1 | 4×4×512 |
Conv-16 | 4×4×512 | 3×3×512-1-1 | 4×4×512 |
MaxPool-5 | 4×4×512 | 2×2-2-0 | 2×2×512 |
Dropout5 | 2×2×512 | — | 2×2×512 |
FC-1 | 2×2×512 | — | 500 |
Dropout6 | 1×500 | — | 1×500 |
表情分类 | CK+ | JAFFE |
---|---|---|
Angry(生气) | 45 | 31 |
Disgust(厌恶) | 59 | 29 |
Fear(恐惧) | 25 | 30 |
Happy(高兴) | 69 | 32 |
Sad(悲伤) | 28 | 30 |
Surprise(惊讶) | 83 | 31 |
表3 CK+与JAFFE数据集的6类表情数量分布
Table 3 Number distribution of 6 types of expressions in CK+ and JAFFE datasets
表情分类 | CK+ | JAFFE |
---|---|---|
Angry(生气) | 45 | 31 |
Disgust(厌恶) | 59 | 29 |
Fear(恐惧) | 25 | 30 |
Happy(高兴) | 69 | 32 |
Sad(悲伤) | 28 | 30 |
Surprise(惊讶) | 83 | 31 |
算法 | 识别率/% | |
---|---|---|
CK+ | JAFFE | |
人工特征提取 | 88.14 | 87.32 |
CNN+LBP | 92.08 | 88.33 |
文献[ | 93.68 | 88.73 |
VGG16 | 95.00 | 90.86 |
文献[ | 95.40 | 92.10 |
GL-DCNN | 95.51 | 93.01 |
表4 不同算法在CK+与JAFFE数据集上识别性能比较
Table 4 Comparison of recognition performance of different algorithms on CK+ and JAFFE datasets
算法 | 识别率/% | |
---|---|---|
CK+ | JAFFE | |
人工特征提取 | 88.14 | 87.32 |
CNN+LBP | 92.08 | 88.33 |
文献[ | 93.68 | 88.73 |
VGG16 | 95.00 | 90.86 |
文献[ | 95.40 | 92.10 |
GL-DCNN | 95.51 | 93.01 |
处理器 | 训练时间/s | 识别率/% |
---|---|---|
CPU | 130.00 | 91.69 |
GPU | 2.00 | 92.71 |
表5 使用CPU和GPU的训练时间和识别率比较
Table 5 Comparison of training time and recognition rate using CPU and GPU
处理器 | 训练时间/s | 识别率/% |
---|---|---|
CPU | 130.00 | 91.69 |
GPU | 2.00 | 92.71 |
[1] |
YANG B, CAO J, NI R, et al. Facial expression recognition using weighted mixture deep neural network based on double-channel facial images[J]. IEEE Access, 2018, 6: 4630-4640.
DOI URL |
[2] |
WU B F, LIN C H. Adaptive feature mapping for customizing deep learning based facial expression recognition model[J]. IEEE Access, 2018, 6: 12451-12461.
DOI URL |
[3] | ZHANG T, ZHENG W M, CUI Z, et al. A deep neural network-driven feature learning method for multi-view facial expression recognition[J]. IEEE Transactions on Multime-dia, 2018, 18(12): 2528-2536. |
[4] | ZENG G, ZHOU J, JIA X, et al. Hand-crafted feature guided deep learning for facial expression recognition[C]// Proceedings of the 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition, Xi’an, May 15-19, 2018. Was-hington: IEEE Computer Society, 2018: 423-430. |
[5] | WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[C]// LNCS 11211: Proceedings of the 15th European Conference on Computer Vision, Munich, Sep 8-14, 2018. Cham: Springer, 2018: 564-574. |
[6] |
孙晓, 潘汀. 基于兴趣区域深度神经网络的静态面部表情识别[J]. 电子学报, 2017, 45(5): 1189-1197.
DOI |
SUN X, PAN T. Static facial expression recognition system using ROI deep neural networks[J]. Acta Electronica Sinica, 2017, 45(5): 1189-1197. | |
[7] | WANG X JIN, LIU W, et al. Feature fusion of HOG and WLD for facial expression recognition[C]// Proceedings of the 2013 IEEE/SICE International Symposium on System Inte-gration, Kobe, Dec 15-17, 2013. Piscataway: IEEE, 2013: 227-232. |
[8] |
JIA Q, GAO X, GUO H, et al. Multi-layer sparse represen-tation for weighted LBP-patches based facial expression reco-gnition[J]. Sensors, 2015, 15(3): 6719.
DOI URL |
[9] | WANG L, LI R F, WANG K, et al. Feature representation for facial expression recognition based on FACS and LBP[J]. International Journal of Automation & Computing, 2014, 11(5): 459-468. |
[10] |
ZHOU J, ZHANG S, MEI H, et al. A method of facial exp-ression recognition based on Gabor and NMF[J]. Pattern Recognition and Image Analysis, 2016, 26(1): 119-124.
DOI URL |
[11] |
SILWAL R, ALSADOON A, PRASAD P W C, et al. A novel deep learning system for facial feature extraction by fusing CNN and MB-LBP and using enhanced loss function[J]. Multimedia Tools and Applications, 2020, 79(1): 1-21.
DOI URL |
[12] | SHIN M, KIM M, KWON D S. Baseline CNN structure analysis for facial expression recognition[C]// Proceedings of the 25th IEEE International Symposium on Robot and Human Interactive Communication, New York, Aug 26-31, 2016. Piscataway: IEEE, 2016: 724-729. |
[13] | CONNIE T, AL-SHABI M, CHEAH W P, et al. Facial ex-pression recognition using a hybrid CNN-SIFT aggregator[C]// LNCS 10607: Proceedings of the 11th International Work-shop on Multi-Disciplinary Trends in Artificial Intelligence, Gadong, Nov 20-22, 2017. Cham: Springer, 2017: 139-149. |
[14] |
ZOU G F, FU G X, GAO M L, et al. A new approach for small sample face recognition with pose variation by fusing gabor encoding features and deep features[J]. Multimedia Tools and Applications, 2020, 79(31): 23571-23598.
DOI URL |
[15] | SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv:1409.1556, 2014. |
[16] | 杨恢先, 张翡, 陈永, 等. 结合改进CSLBP和位平面分解的单样本人脸识别[J]. 激光与光电子学进展, 2018, 55(7): 221-228. |
YANG H X, ZHANG F, CHEN Y, et al. Single-sample face recognition combining improved CSLBP and bit-plane de-composition[J]. Progress in Laser and Optoelectronics, 2018, 55(7): 221-228. | |
[17] | PATIL V, SARODE T. Modified CSLBP[J]. International Journal of Electrical and Computer Engineering, 2019, 9(4):2950-2959. |
[18] | 胡渝苹. 基于HOG-CSLBP与深度学习的跨年龄人脸识别算法[J]. 西南师范大学学报(自然科学版), 2020, 45(3): 115-120. |
HU Y P. A cross-age face recognition algorithm based on HOG-CSLBP and deep learning[J]. Journal of Southwest China Normal University (Natural Science Edition), 2020, 45(3): 115-120. | |
[19] | 董夙慧, 徐永刚. 基于HOG-CSLBP的快速行人检测算法[J]. 计算机工程与设计, 2018, 39(4): 1125-1129. |
DONG S H, XU Y G. Fast pedestrian detection algorithm based on HOG-CSLBP[J]. Computer Engineering and Design, 2018, 39(4): 1125-1129. | |
[20] | 翟海庆, 刘丹, 刘晙. 利用双流卷积神经网络的人脸表情识别方法[J]. 光学技术, 2020, 46(6): 712-720. |
ZHAI H Q, LIU D, LIU J. A facial expression recognition method based on dual-stream convolutional neural network[J]. Optical Technique, 2020, 46(6): 712-720. | |
[21] | LUCEY P, COHN J F, KANADE T J, et al. The extended Cohn-Kanade dataset (CK+) a complete dataset for action unit and emotion-specified expression[C]// Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Re-cognition, San Francisco, Jun 13-18, 2010. Washington: IEEE Computer Society, 2010: 94-101. |
[22] | 林克正, 白婧轩, 李昊天, 等. 深度学习下融合不同模型的小样本表情识别[J]. 计算机科学与探索, 2020, 14(3): 482-492. |
LIN K Z, BAI J X, LI H T, et al. Facial expression recognition with small samples fused with different models under deep learning[J]. Journal of Frontiers of Computer Science and Technology, 2020, 14(3): 482-492. | |
[23] |
LOPES A T, DEAGUIAR E, DESOUZA A F, et al. Facial expression recognition with convolutional neural networks: coping with few data and the training sample order[J]. Pattern Recognition, 2017, 61(1): 610-628.
DOI URL |
[24] | 张俞晴, 何宁, 魏润辰. 基于卷积神经网络融合SIFT特征的人脸表情识别[J]. 计算机应用与软件, 2019, 36(11): 161-167. |
ZHANG Y Q, HE N, WEI R C. Face expression recog-nition based on convolutional neural networks fusing SIFT features[J]. Computer Applications and Software, 2019, 36(11): 161-167. |
[1] | 安凤平, 李晓薇, 曹翔. 权重初始化-滑动窗口CNN的医学图像分类[J]. 计算机科学与探索, 2022, 16(8): 1885-1897. |
[2] | 黄浩, 葛洪伟. 强化类间区分的深度残差表情识别网络[J]. 计算机科学与探索, 2022, 16(8): 1842-1849. |
[3] | 曾凡智, 许露倩, 周燕, 周月霞, 廖俊玮. 面向智慧教育的知识追踪模型研究综述[J]. 计算机科学与探索, 2022, 16(8): 1742-1763. |
[4] | 洪惠群, 沈贵萍, 黄风华. 表情识别技术综述[J]. 计算机科学与探索, 2022, 16(8): 1764-1778. |
[5] | 刘艺, 李蒙蒙, 郑奇斌, 秦伟, 任小广. 视频目标跟踪算法综述[J]. 计算机科学与探索, 2022, 16(7): 1504-1515. |
[6] | 赵小明, 杨轶娇, 张石清. 面向深度学习的多模态情感识别研究进展[J]. 计算机科学与探索, 2022, 16(7): 1479-1503. |
[7] | 彭豪, 李晓明. 多尺度选择金字塔网络的小样本目标检测算法[J]. 计算机科学与探索, 2022, 16(7): 1649-1660. |
[8] | 夏鸿斌, 肖奕飞, 刘渊. 融合自注意力机制的长文本生成对抗网络模型[J]. 计算机科学与探索, 2022, 16(7): 1603-1610. |
[9] | 赵运基, 范存良, 张新良. 融合多特征和通道感知的目标跟踪算法[J]. 计算机科学与探索, 2022, 16(6): 1417-1428. |
[10] | 孙方伟, 李承阳, 谢永强, 李忠博, 杨才东, 齐锦. 深度学习应用于遮挡目标检测算法综述[J]. 计算机科学与探索, 2022, 16(6): 1243-1259. |
[11] | 刘雅芬, 郑艺峰, 江铃燚, 李国和, 张文杰. 深度半监督学习中伪标签方法综述[J]. 计算机科学与探索, 2022, 16(6): 1279-1290. |
[12] | 童敢, 黄立波. Winograd快速卷积相关研究综述[J]. 计算机科学与探索, 2022, 16(5): 959-971. |
[13] | 钟梦圆, 姜麟. 超分辨率图像重建算法综述[J]. 计算机科学与探索, 2022, 16(5): 972-990. |
[14] | 裴利沈, 赵雪专. 群体行为识别深度学习方法研究综述[J]. 计算机科学与探索, 2022, 16(4): 775-790. |
[15] | 赵鹏飞, 谢林柏, 彭力. 融合注意力机制的深层次小目标检测算法[J]. 计算机科学与探索, 2022, 16(4): 927-937. |
阅读次数 | ||||||
全文 |
|
|||||
摘要 |
|
|||||