Survey on Application of Homomorphic Encryption in Deep Learning

doi:10.3778/j.issn.1673-9418.2406098

Abstract

Abstract: With the widespread application of deep learning in various fields, data privacy and security issues have become increasingly important. Homomorphic encryption, a technique that allows computations to be performed directly on encrypted data, offers a potential solution to these problems. This paper surveys methods that combine deep learning with homomorphic encryption, exploring how to effectively apply deep learning models in encrypted environments. Firstly, the basics of homomorphic encryption are introduced, covering its basic principles, different classifications (including partially homomorphic encryption, somewhat homomorphic encryption and fully homomorphic encryption), and the development history of fully homomorphic encryption. Key models in deep learning, such as convolutional neural network and Transformer, are then detailed. The steps of combining homomorphic encryption with deep learning and how to adapt various layers of deep learning (e.g., convolutional layers, attention layer and activation function layer) to the homomorphic encryption environments are discussed. Subsequently, existing methods that integrate convolutional neural network and Transformer with homomorphic encryption are focused on. Specific implementation schemes for performing deep learning computations on encrypted data and performance optimization strategies employed to enhance efficiency and accuracy are discussed. The advantages and limitations of each method are summarized. Finally, current research progress is summarized, and an outlook on future research directions is provided.

Key words: homomorphic encryption, deep learning, convolutional neural networks, Transformer

摘要： 随着深度学习在各种领域中的广泛应用，数据隐私和安全性问题变得日益重要。同态加密作为一种能够在加密数据上直接进行计算的加密技术，为解决这一问题提供了可能的解决方案。综述了深度学习与同态加密结合的方法，探讨了如何在加密环境中有效应用深度学习模型。介绍了同态加密技术的基础知识，涵盖了其基本原理、不同分类（部分同态加密、有限同态加密、全同态加密）以及全同态加密的发展历程。详细介绍了深度学习中的关键模型，包括卷积神经网络和Transformer模型。探讨了同态加密与深度学习结合的步骤以及如何将深度学习的各个层（如卷积层、注意力层、激活函数层）适配于同态加密环境。重点综述了现有的将卷积神经网络和Transformer与同态加密结合的具体方法，探讨了在加密数据上进行深度学习计算的实现方案以及为了提升效率和精度而采用的性能优化策略，并总结了每种方法的优势和局限性。总结了当前研究的进展，并对未来的研究方向进行了展望。

关键词: 同态加密, 深度学习, 卷积神经网络, Transformer

YANG Hongchao, YI Mengjun, LI Peijia, ZHANG Hanwen, SHEN Furao, ZHAO Jian, WANG Liuwang. Survey on Application of Homomorphic Encryption in Deep Learning[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(12): 3065-3079.

杨洪朝, 易梦军, 李培佳, 张瀚文, 申富饶, 赵健, 王刘旺. 同态加密在深度学习中的应用综述[J]. 计算机科学与探索, 2024, 18(12): 3065-3079.

References

[1] LECUN Y, BENGIO Y, HINTON G. Deep learning[J]. Nature, 2015, 521(7553): 436-444.
[2] YI X, PAULET R, BERTINO E, et al. Homomorphic encryp-tion[M]. Berlin, Heidelberg: Springer, 2014.
[3] LI Z, LIU F, YANG W, et al. A survey of convolutional neural networks: analysis, applications, and prospects[J]. IEEE Tran-sactions on Neural Networks and Learning Systems, 2021, 33(12): 6999-7019.
[4] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems 30, Long Beach, Dec 4-9, 2017: 5998-6008.
[5] ACAR A, AKSU H, ULUAGAC A S, et al. A survey on homo-morphic encryption schemes: theory and implementation[J]. ACM Computing Surveys, 2018, 51(4): 1-35.
[6] RIVEST R L, SHAMIR A, ADLEMAN L. A method for obtaining digital signatures and public-key cryptosystems[J]. Communications of the ACM, 1978, 21(2): 120-126.
[7] ELGAMAL T. A public key cryptosystem and a signature scheme based on discrete logarithms[J]. IEEE Transactions on Information Theory, 1985, 31(4): 469-472.
[8] BONEH D, GOH E J, NISSIM K. Evaluating 2-DNF formulas on ciphertexts[C]//Proceedings of the 2nd Theory of Cryptography Conference, Cambridge, Feb 10-12, 2005. Berlin, Heidelberg: Springer, 2005: 325-341.
[9] GENTRY C. Fully homomorphic encryption using ideal lattices[C]//Proceedings of the 41st Annual ACM Symposium on Theory of computing. New York: ACM, 2009: 169-178.
[10] SMART N P, VERCAUTEREN F. Fully homomorphic encry-ption with relatively small key and ciphertext sizes[C]//Proceedings of the 2010 International Workshop on Public Key Cryptography. Berlin, Heidelberg: Springer, 2010: 420-443.
[11] VAN DIJK M, GENTRY C, HALEVI S, et al. Fully homomorphic encryption over the integers[C]//Proceedings of the 29th Annual International Conference on the Theory and Applications of Cryptographic Techniques, French Riviera, May 30-Jun 3, 2010. Berlin, Heidelberg: Springer, 2010: 24-43.
[12] BRAKERSKI Z, VAIKUNTANATHAN V. Fully homomorphic encryption from ring-LWE and security for key dependent messages[C]//Proceedings of the Annual Cryptology Conference. Berlin, Heidelberg: Springer, 2011: 505-524.
[13] LóPEZ-ALT A, TROMER E, VAIKUNTANATHAN V. On-the-fly multiparty computation on the cloud via multikey fully homomorphic encryption[C]//Proceedings of the 44th Annual ACM Symposium on Theory of Computing. New York: ACM, 2012: 1219-1234.
[14] FAN J, VERCAUTEREN F. Somewhat practical fully homo-morphic encryption[J]. IACR Cryptology ePrint Archive, 2012: 144.
[15] CHEON J H, KIM A, KIM M, et al. Homomorphic encryption for arithmetic of approximate numbers[C]//Proceedings of the 23rd International Conference on the Theory and Applications of Cryptology and Information Security, Hong Kong, China, Dec 3-7, 2017. Cham: Springer, 2017: 409-437.
[16] CHILLOTTI I, GAMA N, GEORGIEVA M, et al. TFHE: fast fully homomorphic encryption over the torus[J]. Journal of Cryptology, 2020, 33(1): 34-91.
[17] CHEON J H, HAN K, KIM A, et al. A full RNS variant of approximate homomorphic encryption[C]//Proceedings of the 25th International Conference on Selected Areas in Cryptography, Calgary, Aug 15-17, 2018. Cham: Springer, 2019: 347-368.
[18] LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278-2324.
[19] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[C]//Advances in Neural Information Processing Systems 25,Lake Tahoe, Dec 3-6, 2012: 1106-1114.
[20] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. [2024-04-13]. https://arxiv.org/abs/1409.1556.
[21] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Washington: IEEE Computer Society, 2016: 770-778.
[22] SALEHINEJAD H, SANKAR S, BARFETT J, et al. Recent advances in recurrent neural networks[EB/OL]. [2024-04-13]. https://arxiv.org/abs/1801.01078.
[23] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understand-ing[EB/OL]. [2024-04-13]. https://arxiv.org/abs/1810.04805.
[24] BROWN T, MANN B, RYDER N, et al. Language models are few-shot learners[C]//Advances in Neural Information Processing Systems 33, Dec 6-12, 2020: 1877-1901.
[25] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: transformers for image recognition at scale[EB/OL]. [2024-04-13]. https://arxiv.org/abs/2010.11929.
[26] GILAD-BACHRACH R, DOWLIN N, LAINE K, et al. Crypto-Nets: applying neural networks to encrypted data with high throughput and accuracy[C]//Proceedings of the 33rd International Conference on Machine Learning, New York, Jun 19-24, 2016: 201-210.
[27] LEE J, LEE E, LEE J W, et al. Precise approximation of convolutional neural networks for homomorphically encry-pted data[J]. IEEE Access, 2023, 11: 62062-62076.
[28] CHEN T, BAO H, HUANG S, et al. The-X: privacy-preserving transformer inference with homomorphic encryption[EB/OL]. [2024-04-13]. https://arxiv.org/abs/2206.00216.
[29] HAO M, LI H, CHEN H, et al. Iron: private inference on transformers[C]//Advances in Neural Information Processing Systems 35, New Orleans, Nov 28-Dec 9, 2022: 15718-15731.
[30] LU W, HUANG Z, GU Z, et al. BumbleBee: secure two-party inference framework for large transformers[J]. IACR Cryptology ePrint Archive, 2023: 1678.
[31] BJORCK N, GOMES C P, SELMAN B, et al. Understanding batch normalization[C]//Advances in Neural Information Processing Systems 31, Montréal, Dec 3-8, 2018: 7705-7716.
[32] BA J L, KIROS J R, HINTON G E. Layer normalization[EB/OL]. [2024-04-13]. https://arxiv.org/abs/1607.06450.
[33] ULYANOV D, VEDALDI A, LEMPITSKY V. Instance nor-malization: the missing ingredient for fast stylization[EB/OL]. [2024-04-13]. https://arxiv.org/abs/1607.08022.
[34] FENG J, YANG L T, ZHU Q, et al. Privacy-preserving tensor decomposition over encrypted data in a federated cloud environment[J]. IEEE Transactions on Dependable and Secure Computing, 2018, 17(4): 857-868.
[35] FENG J, YANG L T, ZHANG R, et al. Privacy preserving high-order bi-lanczos in cloud-fog computing for industrial applications[J]. IEEE Transactions on Industrial Informatics, 2020, 18(10): 7009-7018.
[36] ZHANG P, CHENG X, SU S, et al. Task allocation under geo-indistinguishability via group-based noise addition[J]. IEEE Transactions on Big Data, 2022, 9(3): 860-877.
[37] ZHANG P, CHENG X, SU S, et al. Effective truth discovery under local differential privacy by leveraging noise-aware probabilistic estimation and fusion[J]. Knowledge Based Systems, 2023, 261: 110213.
[38] BOS J W, LAUTER K, LOFTUS J, et al. Improved security for a ring-based fully homomorphic encryption scheme[C]//Proceedings of the 14th IMA International Conference on Cryptography and Coding, Oxford, Dec 17-19, 2013. Berlin, Heidelberg: Springer, 2013: 45-64.
[39] CHABANNE H, DE WARGNY A, MILGRAM J, et al. Privacy-preserving classification on deep neural network[J]. IACR Cryptology ePrint Archive, 2017: 35.
[40] HESAMIFARD E, TAKABI H, GHASEMI M. CryptoDL: deep neural networks over encrypted data[EB/OL]. [2024-04-13]. https://arxiv.org/abs/1711.05189.
[41] SANYAL A, KUSNER M, GASCON A, et al. TAPAS: tricks to accelerate (encrypted) prediction as a service[C]//Procee-dings of the 35th International Conference on Machine Learning, Stockholmsm?ssan, Jul 10-15, 2018: 4497-4506.
[42] JIANG X, KIM M, LAUTER K, et al. Secure outsourced matrix computation and application to neural networks[C]//Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security. New York: ACM, 2018: 1209-1222.
[43] ZHOU J, LI J, PANAOUSIS E, et al. Deep binarized convolutional neural network inferences over encrypted data[C]//Proceedings of the 2020 7th IEEE International Conference on Cyber Security and Cloud Computing/2020 6th IEEE International Conference on Edge Computing and Scalable Cloud. Piscataway: IEEE, 2020: 160-167.
[44] DISABATO S, FALCETTA A, MONGELLUZZO A, et al. A privacy-preserving distributed architecture for deep-learning-as-a-service[C]//Proceedings of the 2020 International Joint Conference on Neural Networks. Piscataway: IEEE, 2020: 1-8.
[45] OBLA S, GONG X, ALOUFI A, et al. Effective activation functions for homomorphic evaluation of deep neural networks[J]. IEEE Access, 2020, 8: 153098-153112.
[46] CHEON J H, KIM D, KIM D. Efficient homomorphic comparison methods with optimal complexity[C]//Proceedings of the 26th International Conference on the Theory and Application of Cryptology and Information Security, Daejeon, Dec 7-11, 2020. Cham: Springer, 2020: 221-256.
[47] AL BADAWI A, JIN C, LIN J, et al. Towards the AlexNet moment for homomorphic encryption: HCNN, the first homo-morphic CNN on encrypted data with GPUs[J]. IEEE Tran-sactions on Emerging Topics in Computing, 2020, 9(3): 1330-1343.
[48] LEE J W, KANG H C, LEE Y, et al. Privacy-preserving machine learning with fully homomorphic encryption for deep neural network[J]. IEEE Access, 2022, 10: 30039-30054.
[49] KIM D, GUYOT C. Optimized privacy-preserving CNN inference with fully homomorphic encryption[J]. IEEE Transactions on Information Forensics and Security, 2023, 18: 2175-2187.
[50] ZHU Y, WANG X, JU L, et al. FxHENN: FPGA-based acceleration framework for homomorphic encrypted CNN inference[C]//Proceedings of the 2023 IEEE International Sym-posium on High-Performance Computer Architecture. Pisca-taway: IEEE, 2023: 896-907.
[51] KIM D, PARK J, KIM J, et al. HyPHEN: a hybrid packing method and its optimizations for homomorphic encryption-based neural networks[J]. IEEE Access, 2024, 12: 3024-3038.
[52] HU Z, CHEN L, WANG Y, et al. A secure convolutional neural network inference model based on homomorphic encryption[C]//Proceedings of the 2024 7th World Conference on Computing and Communication Technologies. Piscataway: IEEE, 2024: 17-23.
[53] BRAKERSKI Z, GENTRY C, VAIKUNTANATHAN V. (Leveled) fully homomorphic encryption without bootstrapping[J]. ACM Transactions on Computation Theory, 2014, 6(3): 1-36.
[54] ZIMERMAN I, BARUCH M, DRUCKER N, et al. Convert-ing transformers to polynomial form for secure inference over homomorphic encryption[EB/OL]. [2024-04-13]. https://arxiv.org/abs/2311.08610.
[55] LIU X, LIU Z. LLMs can understand encrypted prompt: towards privacy-computing friendly transformers[EB/OL]. [2024-04-13]. https://arxiv.org/abs/2305.18396.
[56] ZHENG M, LOU Q, JIANG L. Primer: fast private transformer inference on encrypted data[C]//Proceedings of the 2023 60th ACM/IEEE Design Automation Conference. Piscataway: IEEE, 2023: 1-6.
[57] LEE S, LEE G, KIM J W, et al. HETAL: efficient privacy-preserving transfer learning with homomorphic encryption[C]//Proceedings of the 40th International Conference on Machine Learning, Honolulu, Jul 23-29, 2023: 19010-19035.
[58] ZHANG J, LIU J, YANG X, et al. Secure transformer inference made non-interactive[J]. IACR Cryptology ePrint Archive, 2024: 136.