Face Forgery Detection and Attribution via Prototype Disentanglement

doi:10.3778/j.issn.1673-9418.2401016

Abstract

Abstract: The detection and attribution of face forgery aims to determine whether a face in an image or video has been manipulated or synthesized using Deepfake techniques, as well as to further analyze the Deepfake method behind it. However, existing works often treat these two tasks as separate research directions or adopt a sequential approach of first detecting and then attributing, which is difficult to meet practical business needs. A prototype disentangled-based framework for detection and attribution of face forgery is proposed, aiming to use a unified model to simultaneously perform the face forgery detection and Deepfake attribution. The framework consists of a feature encoder, a set of learnable prototypes, a prototype update module, and a detection-and-attribution module. Firstly, the prototype update module is used to disentangle and learn the real prototypes, forgery-specific prototypes, and forgery-shared prototype within the prototype set. Then, the detection-and-attribution module calculates the distance between the sample’s features and all prototypes, and performs face forgery detection and Deepfake attribution tasks simultaneously. Experimental results on the FaceForensics++ and ForgeryNet datasets demonstrate that the proposed framework achieves better performance than existing face forgery detection and attribution methods.

Key words: face forgery detection, Deepfake attribution, prototype disentanglement

摘要： 虚假人脸检测和生成方法溯源旨在判断图像或视频中人脸是否经过深度伪造技术篡改或合成，并进一步分析虚假人脸背后所使用的深度伪造方法。然而，现有工作多将这两个任务作为独立的研究方向进行探索，或者采用串行的方法先检测再溯源，难以满足实际的业务需求。针对这一问题，提出了一种基于原型解耦的虚假人脸检测和生成方法溯源框架，旨在使用单一模型同时进行虚假人脸真假二分类和深度伪造生成方法溯源任务。该框架由一个特征编码器、一组可学习的原型集合、一个原型更新模块以及一个检测和溯源模块组成。通过原型更新模块解耦并学习原型集合中的真实原型、伪造独有原型和伪造共享原型。由检测和溯源模块计算样本特征与所有原型的相似度，再同步进行虚假人脸检测和深度伪造生成方法溯源任务。在FaceForensics++和ForgeryNet数据集上的实验结果表明，相比现有的虚假人脸检测与生成方法溯源工作，所提出的模型实现了更优的性能。

关键词: 虚假人脸检测, 深度伪造生成方法溯源, 原型解耦

QIAN Fei, LI Wei, CHEN Peng, CHEN Haoran, XIE Lipeng, LIU Liyuan. Face Forgery Detection and Attribution via Prototype Disentanglement[J]. Journal of Frontiers of Computer Science and Technology, 2025, 19(2): 490-501.

钱菲, 李威, 陈鹏, 陈浩然, 谢礼鹏, 刘荔园. 基于原型解耦的虚假人脸检测和生成方法溯源[J]. 计算机科学与探索, 2025, 19(2): 490-501.

References

[1] 费建伟, 夏志华, 余佩鹏, 等. 人脸合成技术综述[J]. 计算机科学与探索, 2021, 15(11): 2025-2047.
FEI J W, XIA Z H, YU P P, et al. Survey of face synthesis[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(11): 2025-2047.
[2] MATERN F, RIESS C, STAMMINGER M. Exploiting visual artifacts to expose deepfakes and face manipulations[C]//Proceedings of the 2019 IEEE Winter Applications of Computer Vision Workshops. Piscataway: IEEE, 2019: 83-92.
[3] LI L Z, BAO J M, ZHANG T, et al. Face X-ray for more general face forgery detection[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 5000-5009.
[4] YANG X, LI Y Z, LYU S W. Exposing deep fakes using inconsistent head poses[C]//Proceedings of the 2019 IEEE International Conference on Acoustics, Speech and Signal Processing. Piscataway: IEEE, 2019: 8261-8265.
[5] QIAN Y Y, YIN G J, SHENG L, et al. Thinking in frequency: face forgery detection by mining frequency-aware clues[C]//Proceedings of the 16th European Conference on Computer Vision. Cham: Springer, 2020: 86-103.
[6] ZHANG B, ZHOU J P, SHUMAILOV I, et al. On attribution of deepfakes[EB/OL].[2023-11-26].https://arxiv.org/abs/2008.09194.
[7] GIRISH S, SURI S, RAMBHATLA S, et al. Towards discovery and attribution of open-world GAN generated images[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2021: 14074-14083.
[8] JIA S, LI X, LYU S W. Model attribution of face-swap deepfake videos[C]//Proceedings of the 2022 IEEE International Conference on Image Processing. Piscataway: IEEE, 2022: 2356-2360.
[9] GUARNERA L, GIUDICE O, NIEßNER M, et al. On the exploitation of deepfake model recognition[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway: IEEE, 2022: 61-70.
[10] SUN Z M, CHEN S, YAO T P, et al. Contrastive pseudo learning for open-world DeepFake attribution[C]//Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2023: 20825-20835.
[11] YU N, DAVIS L, FRITZ M. Attributing fake images to GANs: learning and analyzing GAN fingerprints[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2019: 7555-7565.
[12] YANG T Y, HUANG Z Y, CAO J, et al. Deepfake network architecture attribution[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36(4): 4662-4670.
[13] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: transformers for image recognition at scale[C]//Proceedings of the 8th International Conference on Learning Representations, 2020: 1-21.
[14] RÖSSLER A, COZZOLINO D, VERDOLIVA L, et al. FaceForensics: learning to detect manipulated facial images[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2019: 1-11.
[15] HE Y N, GAN B, CHEN S Y, et al. ForgeryNet: a versatile benchmark for comprehensive forgery analysis[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021: 4358-4367.
[16] LI Y Z, CHANG M C, LYU S W. In ictu oculi: exposing AI created fake videos by detecting eye blinking[C]//Proceedings of the 2018 IEEE International Workshop on Information Forensics and Security. Piscataway: IEEE, 2018: 1-7.
[17] GUO H, HU S, WANG X, et al. Eyes tell all: irregular pupil shapes reveal GAN-generated faces[C]//Proceedings of the 2022 IEEE International Conference on Acoustics, Speech and Signal Processing. Piscataway: IEEE, 2022: 2904-2908.
[18] HU S, LI Y Z, LYU S W. Exposing GAN-generated faces using inconsistent corneal specular highlights[C]//Proceedings of the 2021 IEEE International Conference on Acoustics, Speech and Signal Processing. Piscataway: IEEE, 2021: 2500-2504.
[19] FRANK J, EISENHOFER T, SCHÖNHERR L, et al. Leveraging frequency analysis for deep fake image recognition[C]//Proceedings of the 37th International Conference on Machine Learning. New York: ACM, 2020: 3247-3258.
[20] TAN C C, ZHAO Y, WEI S K, et al. Frequency-aware deepfake detection: improving generalizability through frequency space domain learning[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2024, 38(5): 5052-5060.
[21] CHEN P, LIU J, LIANG T, et al. FSSPOTTER: spotting face-swapped video by spatial and temporal clues[C]//Proceedings of the 2020 IEEE International Conference on Multimedia and Expo. Piscataway: IEEE, 2020: 1-6.
[22] GU Z H, CHEN Y, YAO T P, et al. Delving into the local: dynamic inconsistency learning for DeepFake video detection[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36(1): 744-752.
[23] COZZOLINO D, RÖSSLER A, THIES J, et al. ID-reveal: identity-aware DeepFake video detection[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2021: 15088-15097.
[24] DONG X Y, BAO J M, CHEN D D, et al. Identity-driven DeepFake detection[EB/OL]. [2023-11-26]. https://arxiv.org/abs/2012.03930.
[25] LIU Z Z, QI X J, TORR P H S. Global texture enhancement for fake face detection in the wild[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 8057-8066.
[26] AFCHAR D, NOZICK V, YAMAGISHI J, et al. MesoNet: a compact facial video forgery detection network[C]//Proceedings of the 2018 IEEE International Workshop on Information Forensics and Security. Piscataway: IEEE, 2018: 1-7.
[27] DANG H, LIU F, STEHOUWER J, et al. On the detection of digital face manipulation[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 5780-5789.
[28] ZHAO H Q, WEI T Y, ZHOU W B, et al. Multi-attentional deepfake detection[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021: 2185-2194.
[29] CAO J Y, MA C, YAO T P, et al. End-to-end reconstruction-classification learning for face forgery detection[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 4103-4112.
[30] WANG Y, YU K, CHEN C, et al. Dynamic graph learning with content-guided spatial-frequency relation reasoning for deepfake detection[C]//Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2023: 7278-7287.
[31] DONG S C, WANG J, JI R H, et al. Implicit identity leakage: the stumbling block to improving deepfake detection generalization[C]//Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2023: 3994-4004.
[32] 李颖, 边山, 王春桃, 等. CNN结合Transformer的深度伪造高效检测[J]. 中国图象图形学报, 2023, 28(3): 804-819.
LI Y, BIAN S, WANG C T, et al. CNN and Transformer-coordinated deepfake detection[J]. Journal of Image and Graphics, 2023, 28(3): 804-819.
[33] BA Z J, LIU Q Y, LIU Z G, et al. Exposing the deception: uncovering more forgery clues for deepfake detection[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2024, 38(2): 719-728.
[34] SHIOHARA K, YAMASAKI T. Detecting deepfakes with self-blended images[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 18699-18708.
[35] SUN K, YAO T P, CHEN S, et al. Dual contrastive learning for general face forgery detection[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36(2): 2316-2324.
[36] ZHUANG W Y, CHU Q, TAN Z T, et al. UIA-ViT: unsupervised inconsistency-aware method based on vision transformer for face forgery detection[C]//Proceedings of the 17th European Conference on Computer Vision. Cham: Springer, 2022: 391-407.
[37] GUO Y, ZHEN C, YAN P F. Controllable guide-space for generalizable face forgery detection[C]//Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2023: 20761-20770.
[38] YAN Z Y, ZHANG Y, FAN Y B, et al. UCF: uncovering common features for generalizable deepfake detection[C]//Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2023: 22355-22366.
[39] MARRA F, GRAGNANIELLO D, VERDOLIVA L, et al. Do GANs leave artificial fingerprints?[C]//Proceedings of the 2019 IEEE Conference on Multimedia Information Processing and Retrieval. Piscataway: IEEE, 2019: 506-511.
[40] DING Y Z, THAKUR N, LI B X. Does a GAN leave distinct model-specific fingerprints?[C]//British Machine Vision Conference, 1997.
[41] BUI T, YU N, COLLOMOSSE J. RepMix: representation mixing for robust attribution of synthesized images[C]//Proceedings of the 17th European Conference on Computer Vision. Cham: Springer, 2022: 146-163.
[42] CIFTCI U A, DEMIR I. How do deepfakes move? Motion magnification for deepfake source detection[EB/OL]. [2023-11-26]. https://arxiv.org/abs/2212.14033.
[43] NARAYAN K, AGARWAL H, THAKRAL K, et al. DeePhy: on deepfake phylogeny[EB/OL]. [2023-11-26]. https://arxiv.org/abs/2209.09111.
[44] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems 30, Long Beach, Dec 4-9, 2017: 5998-6008.
[45] CARION N, MASSA F, SYNNAEVE G, et al. End-to-end object detection with transformers[C]//Proceedings of the 16th European Conference on Computer Vision. Cham: Springer, 2020: 213-229.
[46] JAEGLE A, BORGEAUD S, ALAYRAC J B, et al. Perceiver IO: a general architecture for structured inputs & outputs[EB/OL]. [2023-11-26]. https://arxiv.org/abs/2107.14795.
[47] CHANG F, LIN C C, LU C J, et al. Adaptive prototype learning algorithms: theoretical and experimental studies[J]. Journal of Machine Learning Research, 2006, 7(10): 2125-2148.
[48] YANG H M, ZHANG X Y, YIN F, et al. Convolutional prototype network for open set recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(5): 2358-2370.
[49] SNELL J, SWERSKY K, ZEMEL R. Prototypical networks for few-shot learning[C]//Advances in Neural Information Processing Systems 30, Long Beach, Dec 4-9, 2017: 4077-4087.
[50] GANIN Y, USTINOVA E, AJAKAN H, et al. Domain-adversarial training of neural networks[J]. The Journal of Machine Learning Research, 2016, 17(1): 2096-2030.
[51] DENG J K, GUO J, VERVERAS E, et al. RetinaFace: single-shot multi-level face localisation in the wild[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 5202-5211.
[52] TAN M, LE Q. EfficientNet: rethinking model scaling for convolutional neural networks[C]//Proceedings of the 36th International Conference on Machine Learning, 2019: 6105-6114.
[53] VAN DER MAATEN L, HINTON G. Visualizing data using t-SNE[J]. Journal of Machine Learning Research, 2008, 9: 2579-2605.