Journal of Frontiers of Computer Science and Technology ›› 2021, Vol. 15 ›› Issue (10): 1812-1829.DOI: 10.3778/j.issn.1673-9418.2104022

• Surveys and Frontiers • Previous Articles     Next Articles

Review of Knowledge Distillation in Convolutional Neural Network Compression

MENG Xianfa, LIU Fang, LI Guang, HUANG Mengmeng   

  1. National Key Laboratory of Science and Technology on Automatic Target Recognition, National Defense University of Science and Technology, Changsha 410000, China
  • Online:2021-10-01 Published:2021-09-30



  1. 国防科技大学 自动目标识别重点实验室,长沙 410000


In recent years, convolutional neural network (CNN) has made remarkable achievements in many applications in the field of image analysis with its powerful ability of feature extraction and expression. However, the continuous improvement of CNN performance is almost entirely due to the deeper and larger network model. In this case, the deployment of a complete CNN often requires huge memory overhead and high-performance computing units (such as GPU) support. However, there are limitations in the wide application of CNN in embedded devices with limited computing resources and mobile terminals with high real-time requirements. Therefore, CNN urgently needs network lightweight. At present, the main ways to solve the above problems are knowledge distillation, network pruning, parameter quantization, low rank decomposition, lightweight network design, etc. This paper first introduces the basic structure and development process of convolutional neural network, and briefly describes and compares five typical basic methods of network compression. Then, the knowledge distillation methods are combed and summarized in detail, and the different methods are compared experimentally on the CIFAR data set. Furthermore, the current evaluation system of knowledge distillation methods is introduced. The comparative analysis and evaluation of many types of methods are given. Finally, the preliminary thinking on the future development of this technology is given.

Key words: convolutional neural network (CNN), knowledge distillation, neural network compression, lightweight network



关键词: 卷积神经网络(CNN), 知识蒸馏, 神经网络压缩, 轻量化网络