Multivariate Feature Extraction and Channel Feature Reconstruction Cross-Modality Person Re-identification Method

doi:10.3778/j.issn.1673-9418.2501038

Abstract

Abstract: To address the problem of difficult visible-infrared person re-identification matching caused by significant modal differences between visible light images and infrared images, a multivariate feature extraction and channel feature reconstruction cross-modality person re-identification method is proposed. Firstly, the dual-stream ResNeXt50 serves as the backbone network, extracting different sub-network features through channel grouping convolution. This approach mitigates the imbalance in channel numbers between the two modalities, enhances discriminative feature extraction capabilities, and reduces model complexity to prevent overfitting. Secondly, a multi-level feature reconstruction module is designed to reconstruct and fuse features from various stages in the channel dimension. The channel attention mechanism and adaptive weights are employed to emphasize key discriminative features, eliminate redundant information, and boost the identification capacity of model. Finally, a multi-dimensional feature extraction module is constructed to extract shared features across multiple modalities through parallel convolution of multiple branches. The EMA (efficient multi-scale attention module) attention mechanism is utilized to capture details and global information in the image through feature grouping and cross spatial learning methods. Effective spatial and channel features are learnt to enhance the ability of network to learn key pedestrian features in complex scenes. On the SYSU-MM01 dataset in all search mode, the proposed method achieves rank-1 and mAP scores of 75.35% and 72.37%, respectively. In the indoor search mode, the rank-1 and mAP scores reach 83.57% and 86.03%, respectively. On the RegDB dataset in the visible-infrared retrieval mode, the rank-1 and mAP scores are 93.21% and 87.09%, respectively, while in the infrared-visible retrieval mode, the rank-1 and mAP scores are 91.63% and 86.00%, respectively, demonstrating the effectiveness of the method.

Key words: visible-infrared image, person re-identification, attention mechanism, feature extraction

摘要： 针对可见光图像与红外图像模态差异大导致可见光-红外行人重识别匹配困难的问题，提出一种多元特征提取与通道特征重构的跨模态行人重识别方法。采用双流ResNeXt50作为骨干网络，通过通道分组卷积分别提取不同子网络特征，缓解两种模态通道数不平衡的问题，提升判别特征提取能力，降低模型复杂度，避免过拟合；设计多级特征重构模块，对不同阶段特征进行通道维度的重构融合，并利用通道注意力机制和自适应权重增强具有判别力的关键特征，减少冗余信息，增强模型的辨识能力；构建多元特征提取模块，通过多支路并行卷积提取多元跨模态共享特征，并利用EMA注意力机制，通过特征分组及跨空间学习方法，捕捉图像中的细节和全局信息，学习有效空间和通道特征，增强网络对复杂场景下行人关键特征的学习能力。该方法在SYSU-MM01数据集的全景搜索模式下，rank-1和mAP分别达到75.35%和72.37%，室内搜索模式下，rank-1和mAP分别达到83.57%和86.03%；在RegDB数据集的可见-红外检索模式下，rank-1和mAP分别达到93.21%和87.09%，红外-可见检索模式下，rank-1和mAP分别达到91.63%和86.00%，证明了方法的有效性。

关键词: 可见光-红外图像, 行人重识别, 注意力机制, 特征提取

WANG Mingjie, BI Yihan, WANG Rong, LI Chong. Multivariate Feature Extraction and Channel Feature Reconstruction Cross-Modality Person Re-identification Method[J]. Journal of Frontiers of Computer Science and Technology, 2025, 19(10): 2769-2781.

王铭杰, 毕艺瀚, 王蓉, 李冲. 多元特征提取与通道特征重构的跨模态行人重识别方法[J]. 计算机科学与探索, 2025, 19(10): 2769-2781.

References

[1] LUO H, JIANG W, FAN X, et al. A survey on deep learning based person re-identification[J]. Acta Automatica Sinica, 2019, 45(11): 2032-2049.
[2] ZHENG L, YANG Y, HAUPTMANN A G. Person re-identification: past, present and future[EB/OL]. [2024-11-06]. https://arxiv.org/abs/1610.02984.
[3] LIN X Y, LI J X, MA Z Y, et al. Learning modal-invariant and temporal-memory for video-based visible-infrared person re-identification[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 20941-20950.
[4] WU A C, ZHENG W S, YU H X, et al. RGB-infrared cross-modality person re-identification[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway: IEEE, 2017: 5390-5399.
[5] YE M, LAN X Y, LI J W, et al. Hierarchical discriminative learning for visible thermal person re-identification[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2018, 32(1): 7501-7508.
[6] WU Q, DAI P Y, CHEN J, et al. Discover cross-modality nuances for visible-infrared person re-identification[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021: 4330-4339.
[7] HUANG N C, LIU J N, LUO Y J, et al. Exploring modality-shared appearance features and modality-invariant relation features for cross-modality person re-identification[J]. Pattern Recognition, 2023, 135: 109145.
[8] LI D G, WEI X, HONG X P, et al. Infrared-visible cross-modal person re-identification with an X modality[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(4): 4610-4617.
[9] WANG Z X, WANG Z, ZHENG Y Q, et al. Learning to reduce dual-level discrepancy for infrared-visible person re-identification[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2019: 618-626.
[10] YANG B, YE M, CHEN J, et al. Augmented dual-contrastive aggregation learning for unsupervised visible-infrared person re-identification[C]//Proceedings of the 30th ACM International Conference on Multimedia. New York: ACM, 2022: 2843-2851.
[11] LU Y, WU Y, LIU B, et al. Cross-modality person re-identification with shared-specific feature transfer[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 13376-13386.
[12] ZHANG Z Y, JIANG S, HUANG C, et al. RGB-IR cross-modality person ReID based on teacher-student GAN model[J]. Pattern Recognition Letters, 2021, 150: 155-161.
[13] WANG G A, ZHANG T Z, CHENG J, et al. RGB-infrared cross-modality person re-identification via joint pixel and feature alignment[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2019: 3622-3631.
[14] ZHANG Q, LAI C Z, LIU J N, et al. FMCNet: feature-level modality compensation for visible-infrared person re-identification[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 7339-7348.
[15] ZHONG X, LU T Y, HUANG W X, et al. Visible-infrared person re-identification via colorization-based siamese generative adversarial network[C]//Proceedings of the 2020 International Conference on Multimedia Retrieval. New York: ACM, 2020: 421-427.
[16] LI J F, WEN Y, HE L H. SCConv: spatial and channel reconstruction convolution for feature redundancy[C]//Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2023: 6153-6162.
[17] XIE S N, GIRSHICK R, DOLLáR P, et al. Aggregated residual transformations for deep neural networks[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 5987-5995.
[18] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 770-778.
[19] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. [2024-11-06]. https://arxiv.org/abs/1409.1556.
[20] OUYANG D L, HE S, ZHANG G Z, et al. Efficient multi-scale attention module with cross-spatial learning[C]//Proceedings of the 2023 IEEE International Conference on Acoustics, Speech and Signal Processing. Piscataway: IEEE, 2023: 1-5.
[21] YE M, SHEN J B, CRANDALL D J, et al. Dynamic dual-attentive aggregation learning for visible-infrared person re-identification[C]//Proceedings of the 16th European Conference on Computer Vision. Cham: Springer, 2020: 229-247.
[22] YE M, SHEN J B, LIN G J, et al. Deep learning for person re-identification: a survey and outlook[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(6): 2872-2893.
[23] CHEN Y, WAN L, LI Z H, et al. Neural feature search for RGB-infrared person re-identification[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021: 587-597.
[24] HAO X, ZHAO S Y, YE M, et al. Cross-modality person re-identification via modality confusion and center aggregation[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2021: 16383-16392.
[25] YANG M X, HUANG Z Y, HU P, et al. Learning with twin noisy labels for visible-infrared person re-identification[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 14288-14297.
[26] ZHANG Y K, WANG H Z. Diverse embedding expansion network and low-light cross-modality benchmark for visible-infrared person re-identification[C]//Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2023: 2153-2162.
[27] HE L, CHENG D, WANG N, et al. Exploring homogeneous and heterogeneous consistent label associations for unsupervised visible-infrared person ReID[EB/OL]. [2024-11-06].https://arxiv.org/abs/2402.00672.
[28] SHI J, YIN X, CHEN Y, et al. Multi-memory matching for unsupervised visible-infrared person re-identification[C]//Proceedings of the 18th European Conference on Computer Vision. Cham: Springer, 2024: 456-474.
[29] WU R Q, JIAO B L, WANG W X, et al. Enhancing visible-infrared person re-identification with modality- and instance-aware visual prompt learning[C]//Proceedings of the 2024 International Conference on Multimedia Retrieval. New York: ACM, 2024: 579-588.
[30] YU X Y, DONG N, ZHU L H, et al. CLIP-driven semantic discovery network for visible-infrared person re-identification[J]. IEEE Transactions on Multimedia, 2025, 27: 4137-4150.