计算机科学与探索

• 学术研究 •    

多教师对比知识反演的无数据模型压缩方法

林振元,林绍辉,姚益武,何高奇,王长波,马利庄   

  1. 1.华东师范大学 计算机科学与技术学院,上海 200062
    2.北京大学 信息科学技术学院,北京 100871

Multi-teacher Contrastive Knowledge Inversion for Data-free Distillation

LIN Zhenyuan, LIN Shaohui, YAO Yiwu, HE Gaoqi, WANG Changbo, MA Lizhuang   

  1. 1.School of Computer Science and Technology, East China Normal University, Shanghai 200062, China
    2.School of Electronics Engineering and Computer Science, Peking University, Beijing 100871, China

摘要: 知识蒸馏是用于压缩深度神经网络的一种有效方法,但是由于用户数据隐私保护、数据机密性或传输的限制,很多时候人们无法获取到原始数据。现有的无数据知识蒸馏方法仅使用单教师模型进行有偏特征统计,生成的数据和原始数据相比存在着多样性和泛化性差问题,从而导致压缩后模型的准确率不高。为了解决此类问题,本文提出了一种多教师对比知识反演的无数据模型压缩方法,该方法从多个可用的教师模型中提取知识并将其融合到学生模型中,以消除模型有偏统计带来的偏差,增强了合成图片的泛化性。进一步,为防止合成的图像样式单一,本文采用对比学习的策略将当前批次生成的图像与历史的图像进行对比来使得生成器合成与历史不相似的图片。同时,利用生成器和学生网络能够学习不同网络视角下表征的关系,提出教师-学生对比的策略,使得生成器学到的图片拥有更好的表征。通过实验表明,本方法不仅生成视觉上令人满意的图像,而且在多个指标上优于现有的方法。生成的合成图像更接近原始数据集的分布,而且只需要一次生成的图片数据集就能泛化用于不同模型训练。

关键词: 模型压缩, 无数据, 知识蒸馏, 数据保护, 隐私保护

Abstract: Knowledge distillation is an effective method for model compression with access to training data. However, due to privacy, confidentiality, or transmission limitations, people can not get the support of data. Existing data-free knowledge distillation methods only uses biased feature statistics contained in one model and runs into problems with low generalizability and diversity in synthetic images and unsatisfactory student model performance. To address these problems, we proposed a Multi-Teacher Contrastive Knowledge Inversion (MTCKI) method that extracts and fuses model-specific knowledge from the available teacher models into a student model to eliminate model bias. Further, we improve the diversity of synthesized images using contrastive learning, which encourages the synthetic images to be distinguishable from the previously stored images. Then we proposed contrastive loss between multi-teacher and student which encourages the generator and student network to learn the relationship between representations from different network perspectives. Experiments demonstrate that MTCKI not only generates visually satisfactory images but also outperforms existing state-of-the-art approaches. The resulting synthesized images are much closer to the distribution of the original dataset and, therefore, can be generated only once to provide comprehensive guidance for various networks rather than a specific one.

Key words: Model Compression, Data-free, Knowledge Distillation, Data Protection, Privacy-Conscious