Journal of Frontiers of Computer Science and Technology ›› 2022, Vol. 16 ›› Issue (5): 1146-1154.DOI: 10.3778/j.issn.1673-9418.2104106

• Graphics and Image • Previous Articles     Next Articles

Deep Convolutional Neural Network Algorithm Fusing Global and Local Features

CHENG Weiyue1,+(), ZHANG Xueqin2, LIN Kezheng2, LI Ao2   

  1. 1. Heilongjiang College of Business and Technology, Harbin 150025, China
    2. School of Computer Science and Technology, Harbin University of Science and Technology, Harbin 150080, China
  • Received:2021-04-29 Revised:2021-08-05 Online:2022-05-01 Published:2022-05-19
  • About author:CHENG Weiyue, born in 1988, M.S., lecturer. Her research interests include sparse representation, image processing, pattern recognition, etc.
    ZHANG Xueqin, born in 1994, M.S. candidate. Her research interests include image processing, pattern recognition, etc.
    LIN Kezheng, born in 1962, Ph.D., professor, M.S. supervisor. His research interests include image processing, machine vision, pattern recognition, etc.
    LI Ao, born in 1986, Ph.D., associate professor, Ph.D. supervisor. His research interests include sparse representation, image restoration, computer vision, etc.
  • Supported by:
    National Natural Science Foundation of China(62071157);Natural Science Foundation of Heilongjiang Province(F2015040);University Nursing Program for Young Scholars with Creative Talents in Heilongjiang Province(UNPYSCT-2018203)

融合全局与局部特征的深度卷积神经网络算法

程卫月1,+(), 张雪琴2, 林克正2, 李骜2   

  1. 1.黑龙江工商学院,哈尔滨 150025
    2.哈尔滨理工大学 计算机科学与技术学院,哈尔滨 150080
  • 通讯作者: + E-mail: cheng_weiyue@sina.cn
  • 作者简介:程卫月(1988—),女,黑龙江哈尔滨人,硕士,讲师,主要研究方向为稀疏表示、图像处理、模式识别等。
    张雪琴(1994—),女,山西平定人,硕士研究生,主要研究方向为图像处理、模式识别等。
    林克正(1962—),男,山东蓬莱人,博士,教授,硕士生导师,主要研究方向为图像处理、机器视觉、模式识别等。
    李骜(1986—),男,黑龙江哈尔滨人,博士,副教授,博士生导师,主要研究方向为稀疏表示、图像复原、计算机视觉等。
  • 基金资助:
    国家自然科学基金(62071157);黑龙江省自然科学基金(F2015040);黑龙江省青年创新人才项目(UNPYSCT-2018203)

Abstract:

In order to further improve the accuracy of facial expression recognition, a deep convolutional neural network algorithm fusing global and local features (GL-DCNN) is proposed. The algorithm consists of two improved convolutional neural network branches, global branch and local branch, which are used to extract global features and local features respectively. The features of the two branches are weighted and fused, and the fused features are used for classification. Firstly, global features are extracted. The global branch is based on transfer learning, and the improved VGG19 network model is used for feature extraction. Secondly, local features are extracted. In the local branch, central symmetric local binary pattern (CSLBP) algorithm is used for the first feature extraction, and the local texture information of the original image is obtained, which is input into shallow convolutional neural network for the second feature extraction, so that the local features related to facial expressions are automatically extracted. Thirdly, two cascaded fully connected layers are used to reduce the dimension of the features of the two branches, and different weights are assigned to them for weighted fusion. Finally, softmax classifier is used for classification. The experiment is validated on CK+ and JAFFE datasets, and the classification accuracy is over 95% and 93%, respectively. Compared with other five algorithms, this algorithm has a good overall performance, good recognition effect and good robustness, which can provide an effective basis for facial expression recognition.

Key words: facial expression recognition, feature fusion, convolutional neural networks (CNN), deep learning

摘要:

为进一步提高人脸表情识别的准确率,提出一种融合全局与局部特征的深度卷积神经网络算法(GL-DCNN)。该算法由两个改进的卷积神经网络分支组成,全局分支和局部分支,分别用于提取全局特征和局部特征,对两个分支的特征进行加权融合,使用融合后的特征进行分类。首先,提取全局特征,全局分支基于迁移学习,使用改进的VGG19网络模型进行特征提取;其次,提取局部特征,局部分支采用中心对称局部二值模式(CSLBP)算法进行第一次特征提取,得到原始图像的局部纹理信息,将其输入到浅层卷积神经网络进行第二次特征提取,使其自动提取出与表情相关的局部特征;再次,采用两个级联的全连接层对两个分支的特征进行降维,为其分配不同权重,进行加权融合;最后,采用softmax分类器进行分类。实验在CK+和JAFFE数据集上进行验证,分类精度分别达95%以上和93%以上,对比其他五种算法,该算法总体表现较好,具有较好的识别效果和良好的鲁棒性,可为人脸表情识别提供有效依据。

关键词: 表情识别, 特征融合, 卷积神经网络(CNN), 深度学习

CLC Number: