面向扩散大模型的多模态人脸生成方法

doi:10.3778/j.issn.1673-9418.2409063

摘要/Abstract

摘要： 人脸生成是计算机视觉的前沿课题，在刑事侦查、虚拟现实等领域有广泛的应用前景。近年来扩散模型展现出卓越的生成能力，能够根据限定条件生成高语义一致性的图片，应用于生成人脸方向上成为新趋势。然而，现有生成方法中基于常规扩散的方法对条件信息细节理解不足，未能充分利用条件信息精准生成人脸；基于扩散大模型的方法通常需耗费大量计算资源微调模型，抑或添加额外复杂网络且未能均衡融合多模态条件信息。针对上述挑战，提出面向扩散大模型的多模态生成人脸方法MA-adapter，添加小型精简网络提取视觉结构信息融合语义指导扩散大模型精准生成人脸，充分利用扩散大模型生成能力的同时避免耗费大量计算资源进行微调。该模型利用多头注意力模块（MAM）增强图片模态提示，使模型更加关注关键信息；通过多尺度特征模块（MFM）提取多尺度特征信息，为精准指导生成提供保障；设计自适应调节机制（AAM），自适应调节不同特征层的生成指导系数以实现更佳性能。实验结果表明，在MM-CelebA-HQ数据集上与当前主流方法T2I-adapter相比，MA-adapter的感知相似指标LPIPS下降约18.4%，图文匹配指标CLIP-Score提高约13.6%，特征相似指标CLIP-I增长约14.8%。大量实验结果充分验证MA-adapter的有效性及优越性。

关键词: 人脸生成, 多模态, 扩散模型, 智能生成, 注意力机制

Abstract: Face generation represents a cutting-edge topic within the field of computer vision, boasting broad application prospects in areas such as criminal investigation and virtual reality. Recently, diffusion models have exhibited outstanding generative capabilities, capable of producing images with high semantic consistency under specific conditions. Their application in facial generation has become a new trend. However, existing methods based on conventional diffusion inadequately handle the details of conditional information, failing to fully exploit such information for the precise generation of faces. Methods based on large diffusion models typically require significant computational resources to fine-tune the model or necessitate the addition of extra complex networks without achieving a balanced integration of multimodal conditional information. To address these challenges, this study proposes a multimodal generative facial method for diffusion-based large models called MA-adapter. By incorporating a compact auxiliary network to extract visual structural information and integrate semantic guidance, the method aims to harness the generative capabilities of diffusion large models while avoiding the substantial computational resources required for fine-tuning. This model first enhances image modality prompts using a multi-head attention module, focusing more on key information. Subsequently, it extracts multi-scale feature information through a multi-scale feature module, providing guarantees for precise generation guidance. Finally, an adaptive adjustment mechanism is designed to adaptively adjust the generation guidance coefficients of different feature layers to achieve better performance. Experimental results on the MM-CelebA-HQ (multi-modal-CelebA-HQ) dataset show that compared with the current state-of-the-art method T2I-adapter, the perceptual similarity metric LPIPS of the MA-adapter decreases by approximately 18.4%, the image-text matching metric CLIP-Score increases by about 13.6%, and the feature similarity metric CLIP-I grows by approximately 14.8%. Extensive experimental results fully validate the effectiveness and superiority of the MA-adapter.

Key words: face generation, multimodal, diffusion model, intelligent generation, attention mechanism

黄万鑫, 任英杰, 芦天亮, 杨刚, 袁梦娇, 曾高俊. 面向扩散大模型的多模态人脸生成方法[J]. 计算机科学与探索, 2025, 19(10): 2815-2830.

HUANG Wanxin, REN Yingjie, LU Tianliang, YANG Gang, YUAN Mengjiao, ZENG Gaojun. Multimodal Face Generation Method for Diffusion Large Models[J]. Journal of Frontiers of Computer Science and Technology, 2025, 19(10): 2815-2830.

参考文献

[1] HO J, JAIN A, ABBEEL P. Denoising diffusion probabilistic models[C]//Advances in Neural Information Processing Systems 33, 2020: 6840-6851.
[2] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]//Advances in Neural Information Processing Systems 27, 2014: 2672-2680.
[3] NICHOL A, DHARIWAL P, RAMESH A, et al. Glide: towards photorealistic image generatioadnd editing with text-guided diffusion models[EB/OL]. [2024-06-30]. https://arxiv.org/abs/2112.10741.
[4] ROMBACH R, BLATTMANN A, LORENZ D, et al. High-resolution image synthesis with latent diffusion models[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 10674-10685.
[5] RAMESH A, DHARIWAL P, NICHOL A, et al. Hierarchical text-conditional image generation with clip latents[EB/OL]. [2024-06-30]. https://arxiv.org/abs/2204.06125.
[6] RunwayML. Stable diffusion v1.5: stable diffusion v1-5 model card[EB/OL]. (2023-08-24)[2024-06-30]. https://huggingface.co/runwayml/stable-diffusion-v1-5.
[7] 刘泽润, 尹宇飞, 薛文灏, 等. 基于扩散模型的条件引导图像生成综述[J]. 浙江大学学报(理学版), 2023, 50(6): 651-667.
LIU Z R, YIN Y F, XUE W H, et al. A review of conditional image generation based on diffusion models[J]. Journal of Zhejiang University (Science Edition), 2023, 50(6): 651-667.
[8] WANG Z X, ZHANG Z Y, ZHANG X Y, et al. DR2: diffusion-based robust degradation remover for blind face restoration[C]//Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2023: 1704-1713.
[9] KIM M, LIU F, JAIN A, et al. DCFace: synthetic face generation with dual condition diffusion model[C]//Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2023: 12715-12725.
[10] HUANG Z Q, CHAN K C K, JIANG Y M, et al. Collaborative diffusion for multi-modal face generation and editing[C]//Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2023: 6080-6090.
[11] Lambdalabs.sd-image-variations-diffusers: stable diffusion image variations model card[EB/OL]. (2023-02-08)[2024-06-30]. https://huggingface.co/lambdalabs/sd-image-variations-diffusers.
[12] RUIZ N, LI Y Z, JAMPANI V, et al. DreamBooth: fine tuning text-to-image diffusion models for subject-driven generation[C]//Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2023: 22500-22510.
[13] ZHANG L M, RAO A Y, AGRAWALA M. Adding conditional control to text-to-image diffusion models[C]//Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2023: 3813-3824.
[14] MOU C, WANG X T, XIE L B, et al. T2I-adapter: learning adapters to dig out more controllable ability for text-to-image diffusion models[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2024, 38(5): 4296-4304.
[15] HU Y, ZHANG J, LIU S, et al. IP-Adapter: text compatible image prompt adapter for text-to-image diffusion models[EB/OL]. [2024-06-30]. https://arxiv.org/abs/2308.06721.
[16] KARRAS T, AITTALA M, LAINE S, et al. Alias-free generative adversarial networks[C]//Advances in Neural Information Processing Systems 34, 2021: 852-863.
[17] PATASHNIK O, WU Z Z, SHECHTMAN E, et al. StyleCLIP: text-driven manipulation of StyleGAN imagery[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2021: 2065-2074.
[18] RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[C]//Proceedings of the 38th International Conference on Machine Learning. New York: ACM, 2021: 8748-8763.
[19] VAN DEN OORD A, VINYALS O. Neural discrete representation learning[C]//Advances in Neural Information Processing Systems 30, 2017: 6309-6318.
[20] ESSER P, ROMBACH R, OMMER B. Taming transformers for high-resolution image synthesis[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021: 12868-12878.
[21] KIM G, KWON T, YE J C. DiffusionCLIP: text-guided diffusion models for robust image manipulation[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 2416-2425.
[22] SOHL-DICKSTEIN J, WEISS E, MAHESWARANATHAN N, et al. Deep unsupervised learning using nonequilibrium thermodynamics[C]//Proceedings of the 32nd International Conference on Machine Learning, 2015: 2256-2265.
[23] SONG J, MENG C, ERMON S. Denoising diffusion implicit models[EB/OL]. [2024-06-30]. https://arxiv.org/abs/2010.02502.
[24] DHARIWAL P, NICHOL A. Diffusion models beat GANs on image synthesis[C]//Advances in Neural Information Processing Systems 34, 2021: 8780-8794.
[25] BALAJI Y, NAH S, HUANG X, et al. eDiff-I: text-to-image diffusion models with an ensemble of expert denoisers[EB/OL]. [2024-06-30]. https://arxiv.org/abs/2211.01324.
[26] XU X Q, WANG Z Y, ZHANG E, et al. Versatile diffusion: text, images and variations all in one diffusion model[C]//Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2023: 7720-7731.
[27] HUANG, L, CHEN D, LIU Y, et al. Composer: creative and controllable image synthesis with composable conditions[EB/OL]. [2024-06-30]. https://arxiv.org/abs/2302.09778.
[28] HOULSBY N, GIURGIU A, JASTRZEBSKI S, et al. Parameter-efficient transfer learning for NLP[C]//Proceedings of the 36th International Conference on Machine Learning. New York: ACM, 2019: 2790-2799.
[29] ZHAO S, CHEN D, CHEN Y, et al. Uni-ControlNet: all-in-one control to text-to-image diffusion models[C]//Advances in Neural Information Processing Systems 36, 2024.
[30] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems 30, 2017.
[31] WANG Z H, LIU X H, LI H S, et al. CAMP: cross-modal adaptive message passing for text-image retrieval[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2019: 5763-5772.
[32] BORJI A. Generated faces in the wild: quantitative comparison of stable diffusion, midjourney and DALL-E 2[EB/OL]. [2024-06-30]. https://arxiv.org/abs/2210.00586.
[33] SCHUHMANN C, BEAUMOUNT R, VENCU R, et al. LAION-5B: an open large-scale dataset for training next generation image-text models[C]//Advances in Neural Information Processing Systems 35, 2022: 25278-25294.
[34] RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]//Proceedings of the 2015 International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2015: 234-241.
[35] LOSHCHILOV I, HUTTER F. Decoupled weight decay regularization[EB/OL]. [2024-06-30]. https://arxiv.org/abs/1711.05101.
[36] LEE C H, LIU Z, WU L, et al. MaskGAN: towards diverse and interactive facial image manipulation[C]//Proceedings of the 33rd IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 5549-5558.
[37] ZHANG R, ISOLA P, EFROS A A, et al. The unreasonable effectiveness of deep features as a perceptual metric[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 586-595.
[38] HESSEL J, HOLTZMAN A, FORBES M, et al. CLIPScore: a reference-free evaluation metric for image captioning[EB/OL]. [2024-06-30]. https://arxiv.org/abs/2104.08718.
[39] KRIZHEVSKY A, SUTSKERVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[C]//Advances in Neural Information Processing Systems 25, 2012.
[40] CompVis, stable-diffusion-v1-4: stable diffusion v1-4 model card[EB/OL]. (2023-08-24)[2024-06-30]. https://huggingface.co/CompVis/stable-diffusion-v1-4.
[41] Imagepipeline, Realistic-Vision-V6.0： Realistic-Vision-V6.0[EB/OL]. (2024-01-13)[2024-06-30]. https://huggingface.co/imagepipeline/Realistic-Vision-V6.0.
[42] Stablediffusionapi, anything-v5: anything V5 API inference [EB/OL]. (2023-07-05)[2024-06-30]. https://huggingface.co/stablediffusionapi/anything-v5.