Construction and Matching of Thermal Feature Descriptor for Image Retrieval

doi:10.3778/j.issn.1673-9418.2105001

Abstract

Abstract: Image matching technology based on deep learning has become a research hotspot in image retrieval tasks. Affected by the high similarity descriptors of different local regions, the mismatch problem reduces the retrieval accuracy of the existing methods. For this reason, a method of constructing and matching thermal feature descriptors for image retrieval is proposed. Firstly, the feature map of each layer is obtained through the semantic segmentation network. By observing the feature map, it is found that the last convolutional layer has the characteristics of stronger spatial and semantic information. For this feature, the gradient score is used to obtain the weight of each channel in the feature map. Through linear fusion, the channel dimension is weighted and summed and normalized, and the final heat map is obtained by bilinear interpolation. Secondly, the deep local features (DELF) algorithm is used to obtain the image depth feature descriptor. The category information and thermal value information obtained by the semantic segmentation network are used to construct a multi-dimensional composite depth feature descriptor, and the KD tree structure for this type of feature descriptor is given. Finally, based on the structure, it combines BBF (best bin first) and random sampling consensus algorithm to achieve feature matching. Experiments on the Oxford5K and Paris6K public datasets show that the proposed method is better in precision and time compared with the DELF and D2-Net algorithm. Compared with Fine-tuning CNN, DAME WEB and other methods, the retrieval accuracy is improved by nearly 2 percentage points. The method in this paper can better improve the efficiency and accuracy of image retrieval, and the experimental results verify the effectiveness of the method in this paper.

Key words: image retrieval, semantic segmentation network, deep feature descriptor, heat map, KD tree

摘要： 基于深度学习的图像匹配技术已成为图像检索任务的研究热点，受不同局部区域高相似性描述子影响，误匹配问题降低了现有方法的检索精度，为此提出一种面向图像检索的热力特征描述子构造与匹配方法。首先，通过语义分割网络获得每个卷积层的可视化特征图，针对最后一个卷积层具有更强空间信息和语义信息的特点，利用梯度得分获得特征图中每个通道的权重，通过线性融合的方式，在通道维度上加权求和并归一化，利用双线性插值获得最终的热力图。其次，采用深度学习方法（DELF）获得图像深度特征描述子，利用语义分割网络所获得的类别信息和热力值信息构造多维度复合热力特征描述子，给出针对这类特征描述子的KD树结构。最后，基于该结构结合BBF和随机抽样一致算法实现特征匹配。在Oxford5K和Paris6K数据集上进行实验，实验结果表明，与DELF和D2-Net算法相比，该方法在查准率及时间效率上都有所提高，与Fine-tuning CNN、DAME WEB等方法相比检索精度提高近2个百分点。该方法能够更好地提高图像检索效率和精度，实验结果验证了该方法的有效性。

关键词: 图像检索, 语义分割网络, 深度特征描述子, 热力图, KD树

LIU Tianyu, JIA Di, LUO Shunli, WANG Kai. Construction and Matching of Thermal Feature Descriptor for Image Retrieval[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(1): 166-178.

刘天宇, 贾迪, 骆顺利, 王凯. 面向图像检索的热力特征描述子构造与匹配[J]. 计算机科学与探索, 2023, 17(1): 166-178.

References

[1] 杨化超. 图像局部不变性特征及其匹配问题研究与应用[M]. 北京：测绘出版社, 2013.
YANG H C. Research and application of local invariant image features and matching problems[M]. Beijing: Surveying and Mapping?Publishing?House, 2013.
[2] NISTéR D, STEWéNIUS H. Scalable recognition with a vocabulary tree[C]//Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recog-nition, New York, Jun 17-22, 2006. Washington: IEEE Computer Society, 2006: 2161-2168.
[3] BAY H, ESS A, TUYTELAARS T, et al. Speeded-up robust features (SURF)[J]. Computer Vision and Image Under-standing, 2008, 110(3): 346-359.
[4] RUBLEE E, RABAUD V, KONOLIGE K, et al. ORB: an efficient alternative to SIFT or SURF[C]//Proceedings of the 2011 International Conference on Computer vision, Barce- lona, Nov 6-13, 2011. Washington: IEEE Computer Society, 2011: 2564-2571.
[5] LEUTENEGGER S, CHLI M, SIEGWART R Y. BRISK: binary robust invariant scalable keypoints[C]//Proceedings of the 2011 International Conference on Computer Vision,Barcelona, Nov 6-13, 2011. Washington: IEEE Computer Society, 2011: 2548-2555.
[6] CHOI S G, HAN S W. New binary descriptors based on BRISK sampling pattern for image retrieval[C]//Proceedings of the 2014 International Conference on Information and Communication Technology Convergence, Busan, Oct 22-24, 2014. Washington: IEEE Computer Society, 2014: 575-576.
[7] DANG Q B, LE V P, LUQMAN M M, et al. Camera-based document image retrieval system using local features - comparing SRIF with LLAH, SIFT, SURF and ORB[C]//Proceedings of the 2015 13th International Conference on Document Analysis and Recognition, Tunis, Aug 23-26, 2015. Washington: IEEE Computer Society, 2015: 1211-1215.
[8] YI K M, TRULLS E, LEPETIT V, et al. Lift: learned invariant feature transform[C]//LNCS 9910: Proceedings of the 14th European Conference on Computer Vision, Amsterdam, Oct 8-16, 2016. Cham: Springer, 2016: 467-483.
[9] ZAGORUYKO S, KOMODAKIS N. Learning to compare image patches via convolutional neural networks[C]//Pro-ceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, Jun 7-12, 2015.Washington: IEEE Computer Society, 2015: 4353-4361.
[10] GORDO A, ALMAZAN J, REVAUD J, et al. Deep image retrieval: learning global representations for image search[C]//LNCS 9910: Proceedings of the 14th European Conference on Computer Vision, Amsterdam, Oct 8-16, 2016. Cham: Springer, 2016: 241-257.
[11] 洪睿, 康晓东, 李博, 等. 动态匹配核函数图像检索[J]. 中国图象图形学报, 2018, 23(12): 1874-1885.
HONG R, KANG X D, LI B, et al. Application of dynamic match kernel in image retrieval[J]. Journal of Image and Graphics, 2018, 23(12): 1874-1885.
[12] 周书仁, 谢盈, 蔡碧野. 融合多尺度特征的深度哈希图像检索方法[J]. 计算机科学与探索, 2018, 12(12): 1974-1986.
ZHOU S R, XIE Y, CAI B Y. Deep Hashing method for image retrieval based on multi-scale features[J]. Journal of Frontiers of Computer Science and Technology, 2018, 12(12): 1974-1986.
[13] NOH H, ARAUJO A, SIM J, et al. Large-scale image retrieval with attentive deep local features[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Oct 22-29, 2017. Washington: IEEE Computer Society, 2017: 3456-3465.
[14] YANG T Y, KIEN NGUYEN D, HEIJNEN H, et al. DAME WEB: dynamic mean with whitening ensemble binarization for landmark retrieval without human annotation[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision Workshops, Seoul, Oct 27-28, 2019. Piscataway: IEEE, 2019: 2913-2922.
[15] RADENOVIC F, TOLIAS G, CHUM O. Fine-tuning CNN image retrieval with no human annotation[J]. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 2019, 41(7): 1655-1668.
[16] WANG F, JIANG M Q, QIAN C, et al. Residual attention network for image classification[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 6450-6458.
[17] LIU T, YU S, XU B, et al. Recurrent networks with attention and convolutional networks for sentence representation and classification[J].?Applied Intelligence, 2018,?48(10): 3797-3806.
[18] OUYANG D, ZHANG Y, SHAO J. Video-based person re-identification via spatio-temporal attentional and two-stream fusion convolutional networks[J].?Pattern Recognition Letters, 2019,?117: 153-160.
[19] ZHAN H, SH B, KOT A C. Cross-domain shoe retrieval with a semantic hierarchy of attribute classification network[J].?IEEE Transactions on Image Processing,?2017, 26(12): 5867-5881.
[20] XU J, SHI C Z, QI C Z, et al. Unsupervised part-based weighting aggregation of deep convolutional features for image retrieval[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence, the 30th Innovative Applications of Artificial Intelligence, and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence, New Orleans, Feb 2-7, 2018. Menlo Park: AAAI, 2018: 7436-7443.
[21] FURUTA R, INOUE N, YAMASAKI T. Efficient and interactive spatial-semantic image retrieval[J].?Multimedia Tools and Applications,?2019, 78(13): 18713-18733.
[22] BADRINARAYANAN V, KENDALL A, CIPOLLA R. SegNet: a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Ttransactions on Pattern Analysis and Machine Intelligence, 2017, 39(12): 2481-2495.
[23] RAGURAM R, CHUM O, POLLEFEYS M, et al. USAC: a universal framework for random sample consensus[J].?IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012,?35(8): 2022-2038.
[24] YOSINSK J, CLUNE J, NGUYEN A, et al. Understanding neural networks through deep visualization[J]. arXiv:1506. 06579, 2015.
[25] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Con-ference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 770-778.
[26] BUDDEMEIER U, NEVEN H. Systems and methods for descriptor vector computation: U.S. Patent 8098938[P]. 2012-01-17.
[27] DUSMANU M, ROCCO I, PAJDLA T, et al. D2-Net: a trainable CNN for joint description and detection of local features[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach,Jun 16-20, 2019. Washington: IEEE Computer Society, 2019: 8092-8101.
[28] PHILBIN J, CHUM O, ISARD M, et al. Object retrieval with large vocabularies and fast spatial matching[C]//Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, Jun 17-22, 2007. Washington: IEEE Computer Society, 2007: 1-8.
[29] PHILBIN J, CHUM O, ISARD M, et al. Lost in quantization: improving particular object retrieval in large scale image databases[C]//Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, Jun 23-28, 2008. Washington: IEEE Computer Society, 2008: 1-8.
[30] ZHENG Y T, ZHAO M, SONG Y, et al. Tour the world: building a web-scale landmark recognition engine[C]//Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, Jun 20-25, 2009.Washington: IEEE Computer Society, 2009: 1085-1092.