引入激活加权策略的分组排序学习方法

doi:10.3778/j.issn.1673-9418.2101045

计算机科学与探索 ›› 2022, Vol. 16 ›› Issue (7): 1594-1602.DOI: 10.3778/j.issn.1673-9418.2101045

引入激活加权策略的分组排序学习方法

李玉轩¹^,²^,⁺(), 洪学海¹^,³, 汪洋¹, 唐正正¹^,², 班艳¹

1.中国科学院计算机网络信息中心,北京 100190
2.中国科学院大学,北京 100049
3.中国科学院计算技术研究所,北京 100190

收稿日期:2021-01-11 修回日期:2021-03-16 出版日期:2022-07-01 发布日期:2021-04-09
作者简介:李玉轩（1996—）,男,江苏宿迁人,硕士研究生,主要研究方向为机器学习、信息检索等。
LI Yuxuan, born in 1996, M.S. candidate. His research interests include machine learning, information retrieval, etc.
洪学海（1967—）,男,安徽巢湖人,博士,研究员,博士生导师,CCF杰出会员,主要研究方向为高性能计算、人工智能、大数据等。
HONG Xuehai, born in 1967, Ph.D., professor, Ph.D. supervisor, outstanding member of CCF. His research interests include high performance computing, a.pngicial intelligence, big data, etc.
汪洋（1984—）,男,湖北武汉人,博士,高级工程师,硕士生导师,中国科学院计算机网络信息中心信息化发展战略与评估中心主任,主要研究方向为大数据分析、态势感知系统、信息化发展战略研究等。
WANG Yang, born in 1984, Ph.D., senior engineer, M.S. supervisor. His research interests include big data analysis, situational awareness system, strategy research on informatization development, etc.
唐正正（1992—）,男,安徽怀远人,博士研究生,主要研究方向为机器学习、数据挖掘、图表示学习等。
TANG Zhengzheng, born in 1992, Ph.D. candidate. His research interests include machine learning, data mining, graph representation learning, etc.
班艳（1981—）,女,北京人,高级工程师,主要研究方向为信息化发展战略研究。
BAN Yan, born in 1981, senior engineer. Her research interest is strategy research on informatization development.

Groupwise Learning to Rank Algorithm with Introduction of Activated Weighting

LI Yuxuan¹^,²^,⁺(), HONG Xuehai¹^,³, WANG Yang¹, TANG Zhengzheng¹^,², BAN Yan¹

1. Computer Network Information Center, China Academy of Sciences, Beijing 100190, China
2. University of China Academy of Sciences, Beijing 100049, China
3. Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China

Received:2021-01-11 Revised:2021-03-16 Online:2022-07-01 Published:2021-04-09

摘要/Abstract

摘要：

排序学习（LtR）将有监督机器学习技术（SML）用于解决排序问题,旨在给出输入文档列表的相关度更优化的排序结果。此前关于深度排序模型的研究,对于列表内文档的相关度计算彼此独立,缺乏考虑文档之间的相互作用。近年来一些新方法致力于挖掘文档之间的相互影响,如分组评分法（GSF）,通过学习多元变量评分函数来联合判断文档相关性,但大多忽略了文档间相互影响的差异性,同时增加了很大的计算代价。针对此问题,提出了一种带权重的分组深度排序模型（W-GSF）。该方法借鉴推荐领域的深度兴趣网络,引入其根据候选商品调整历史行为序列权重的思想,在排序学习中多元评分法基础上,以多层前馈神经网络为主体结构,并在输入端加入激活单元,利用神经网络自适应学习调整输入的多元变量的权重,来挖掘交叉文档关系的差异性。在公共基准数据集MSLR上的实验验证了该方法的有效性,相比基线排序模型,激活策略的引入带来了排序指标上的明显提升,同时相对于同等效果的排序方法计算量大幅降低。

关键词: 排序学习（LtR）, 分组评分法（GSF）, 深度神经网络, 深度兴趣网络

Abstract:

Learning to rank (LtR) applies supervised machine learning (SML) technologies to the ranking problems, aiming at optimizing the relevance of input document list. As regard to previous studies on the deep ranking model, the calculation of the relevance of the documents in the list is independent of each other, which lacks consideration of document interactions. In recent years, some new methods are devoted to mining the interaction between documents, such as groupwise scoring function (GSF), which learns multivariate scoring function to jointly judge the correlation, but most of these methods ignore the differences of the interaction between documents, and bring high calculation cost at the same time. In order to solve this problem, this paper proposes a weighted groupwise deep ranking model (W-GSF). In view of the deep interest network in the field of recommendation, this paper intro-duces the idea of adjusting the weight of historical behavior sequence according to the candidate products. On the basis of multivariate scoring method in learning to rank field, this method uses muti-layer feed forword neural networks as main structure, and adds an activation unit into it before the input module, taking advantage of neural networks to adjust the weight of input multiple variables adaptively, so as to mine the differences of cross document relationship. Experiments on the public benchmark dataset MSLR verify the effectiveness of the method. Compared with baseline ranking models, the introduction of activation strategy brings a significant improvement of ranking metrics, and the computational complexity is greatly reduced compared with the same effect learning to rank methods.

Key words: learning to rank (LtR), groupwise scoring function (GSF), deep neural network, deep interest network

中图分类号:

TP391

李玉轩, 洪学海, 汪洋, 唐正正, 班艳. 引入激活加权策略的分组排序学习方法[J]. 计算机科学与探索, 2022, 16(7): 1594-1602.

LI Yuxuan, HONG Xuehai, WANG Yang, TANG Zhengzheng, BAN Yan. Groupwise Learning to Rank Algorithm with Introduction of Activated Weighting[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(7): 1594-1602.

图/表 5

图1 基本的GSF模型结构（以只含有三个文档的列表为例）

Fig.1 Structure of basic GSF model, with a sample which only has three documents

图2 W-GSF与激活单元的结构

Fig.2 Structure of W-GSF and activation unit

表1 基线排序学习方法

Table 1 Baseline learning-to-rank models

方法	具体描述
RankingSVM	经典的基于支持向量机的文档对排序方法
LambdaMART	基于树的方法,善于处理稠密特征向量,多个公开数据集上的SOTA技术
RankNet	经典的文档对方法,基于神经网络模型,可以在损失函数上乘上 $λ$ -权重变成文档列表法
GSF-m	群组大小为 $m$ 的分组文档法

表1 基线排序学习方法

Table 1 Baseline learning-to-rank models

方法	具体描述
RankingSVM	经典的基于支持向量机的文档对排序方法
LambdaMART	基于树的方法,善于处理稠密特征向量,多个公开数据集上的SOTA技术
RankNet	经典的文档对方法,基于神经网络模型,可以在损失函数上乘上 $λ$ -权重变成文档列表法
GSF-m	群组大小为 $m$ 的分组文档法

表2 MSLR数据集上模型的NDCG指标对比

Table 2 Comparison of models’ test NDCG on MSLR dataset %

Dataset	Model	NDCG@1	NDCG@3	NDCG@5	NDCG@10
Web10K	RankingSVM	36.22(±0.12)	—	34.91(±0.09)	31.07(±0.09)
	RankNet	39.74(±0.05)	—	40.94(±0.07)	43.37(±0.05)
	LambdaMART	46.20(±0.13)	—	46.23(±0.09)	48.33(±0.11)
	GSF-2	41.79(±0.14)	42.13(±0.09)	42.31(±0.11)	44.79(±0.09)
	GSF-64	42.75(±0.14)	42.61(±0.07)	42.65(±0.12)	44.87(±0.13)
	W-GSF	42.52(±0.09)	42.57(±0.09)	42.70(±0.09)	44.91(±0.09)
Web30K	RankingSVM	37.33(±0.09)	—	35.07(±0.13)	32.61(±0.17)
	RankNet	40.74(±0.11)	—	42.12(±0.05)	44.73(±0.06)
	LambdaMART	50.33(±0.07)	—	49.20(±0.09)	51.05(±0.11)
	GSF-2	43.37(±0.11)	43.41(±0.09)	43.69(±0.11)	46.22(±0.09)
	GSF-64	44.43(±0.09)	44.52(±0.13)	44.89(±0.12)	46.84(±0.10)
	W-GSF	44.41(±0.07)	44.54(±0.09)	44.77(±0.09)	46.88(±0.09)

表3 W-GSF与GSF在MSLR-Web30K上的计算量对比

Table 3 Comparison of calculational cost of W-GSF and GSF on MSLR-Web30K

Model	$Δ$ FLOPs	$Δ$ NDCG@10/%
GSF-2	—	—
GSF-64	×28.10	+1.34
W-GSF	×1.33	+1.43

表3 W-GSF与GSF在MSLR-Web30K上的计算量对比

Table 3 Comparison of calculational cost of W-GSF and GSF on MSLR-Web30K

Model	$Δ$ FLOPs	$Δ$ NDCG@10/%
GSF-2	—	—
GSF-64	×28.10	+1.34
W-GSF	×1.33	+1.43

参考文献 22

[1]	LIU T Y. Learning to rank for information retrieval[C]// Pro-ceeding of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval,Geneva, Jul 19-23, 2010. New York: ACM, 2010: 904.
[2]	CHEN W, LIU T Y, LAN Y Y, et al. Ranking measures and loss functions in learning to rank[C]// Advances in Neural Information Processing Systems 22, Vancouver, Dec 7-10,2009. Red Hook: Curran Associates, 2009: 315-323.
[3]	BURGES C J C. From RankNet to LambdaRank to Lambda-MART: an overview: MSR-TR-2010-82[R]. 2010.
[4]	CAO Z, QIN T, LIU T Y, et al. Learning to rank: from pair-wise approach to listwise approach[C]// Proceedings of the 24th International Conference on Machine Learning, Cor-vallis, Jun 20-24, 2007. New York: ACM, 2007: 129-136.
[5]	曹军梅, 马乐荣. 卷积重提取特征的文档列表排序学习方法[J]. 中文信息学报, 2020, 34(8): 86-93.
	CAO J M, MA L R. Listwise reranking via convolutional re-extracted features[J]. Journal of Chinese Information Pro-cessing, 2020, 34(8): 86-93.
[6]	WANG R X, FANG K, ZHOU R K, et al. SERank: optimize sequencewise learning to rank using squeeze-and-excitation network[J]. arXiv: 2006. 04084, 2020.
[7]	AI Q Y, WANG X H, BRUCH S, et al. Learning groupwise multivariate scoring functions using deep neural networks[C]// Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval, Santa Clara, Oct 2-5, 2019. New York: ACM, 2019: 85-92.
[8]	YE P, DOERMANN D. Combining preference and absolute judgements in a crowd-sourced setting[C]// Proceedings of the 2013 ICML Workshop on Machine Learning Meets Crowd-sourcing, Atlanta, Jun 16-21, 2013: 1-7.
[9]	JOACHIMS T, GRANKA L A, PAN B, et al. Accurately in-terpreting clickthrough data as implicit feedback[C]// Procee-dings of the 28th Annual International ACM SIGIR Con-ference on Research and Development in Information Re-trieval, Salvador, Aug 15-19, 2005. New York: ACM, 2005:154-161.
[10]	ZHOU G R, ZHU X Q, SONG C R, et al. Deep interest network for click-through rate prediction[C]// Proceedings of the 24th ACM SIGKDD International Conference on Know-ledge Discovery and Data Mining, London, Aug 19-23, 2018. New York: ACM, 2018: 1059-1068.
[11]	JOACHIMS T. Training linear SVMs in linear time[C]// Proceedings of the 12th ACM SIGKDD International Con-ference on Knowledge Discovery and Data Mining, Phila-delphia, Aug 20-23, 2006. New York: ACM, 2006: 217-226.
[12]	FRIEDMAN J H. Greedy function approximation: a gra-dient boosting machine[J]. Annals of Statistics, 2001, 29(5):1189-1232. DOI URL
[13]	JOACHIMS T. Optimizing search engines using clickthrough data[C]// Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Phila-delphia, Aug 20-23, 2006. New York: ACM, 2006: 133-142.
[14]	熊李艳, 陈晓霞, 钟茂生, 等. 基于PairWise排序学习算法研究综述[J]. 科学技术与工程, 2017, 17(21): 184-190.
	XIONG L Y, CHEN X X, ZHONG M S, et al. Survey on PairWise of the learning to rank[J]. Science Technology and Engineering, 2017, 17(21): 184-190.
[15]	TAYLOR M J, GUIVER J, ROBERTSON S, et al. SoftRank: optimizing non-smooth rank metrics[C]// Proceedings of the 2008 International Conference on Web Search and Web Data Mining, Palo Alto, Feb 11-12, 2008. New York: ACM, 2008: 77-86.
[16]	黄震华, 张佳雯, 田春岐, 等. 基于排序学习的推荐算法研究综述[J]. 软件学报, 2016, 27(3): 691-713.
	HUANG Z H, ZHANG J W, TIAN C Q, et al. Survey on learning-to-rank based recommendation algorithms[J]. Journal of Software, 2016, 27(3): 691-713.
[17]	AI Q Y, BI K P, GUO J F, et al. Learning a deep listwise context model for ranking refinement[C]// Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, Ann Arbor, Jul 8-12, 2018. New York: ACM, 2018: 135-144.
[18]	HE K M, ZHANG X Y, REN S Q, et al. Delving deep into re.pngiers: surpassing human-level performance on imagenet classification[C]// Proceedings of the 2015 IEEE International Conference on Computer Vision, Santiago, Dec 7-13, 2015. Washington: IEEE Computer Society, 2015: 1026-1034.
[19]	QIN T, LIU T Y. Introducing LETOR 4.0 datasets[J]. arXiv:1306. 2597, 2013.
[20]	JOACHIMS T. A support vector method for multivariate performance measures[C]// Proceedings of the 22nd Inter-national Conference on Machine Learning, Bonn, Aug 7-11, 2005. New York: ACM, 2005: 377-384.
[21]	KE G L, MENG Q, FINLEY T, et al. LightGBM: a highly efficient gradient boosting decision tree[C]// Advances in Neural Information Processing Systems 30, Long Beach, Dec 4-9, 2017. Red Hook: Curran Associates, 2017: 3146-3154.
[22]	BRUCH S, WANG X H, BENDERSKY M, et al. An analysis of the softmax cross entropy loss for learning-to-rank with binary relevance[C]// Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval, Santa Clara, Oct 2-5, 2019. New York: ACM, 2019: 75-78.

引入激活加权策略的分组排序学习方法

Groupwise Learning to Rank Algorithm with Introduction of Activated Weighting

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 5

参考文献 22

相关文章 11

编辑推荐

Metrics

[1]	裴利沈, 赵雪专. 群体行为识别深度学习方法研究综述[J]. 计算机科学与探索, 2022, 16(4): 775-790.
[2]	刘利平, 乔乐乐, 蒋柳成. 图像去噪方法概述[J]. 计算机科学与探索, 2021, 15(8): 1418-1431.
[3]	武晓栋, 刘敬浩, 金杰, 毛思平. 基于DT及PCA的DNN入侵检测模型[J]. 计算机科学与探索, 2021, 15(8): 1450-1458.
[4]	沈学利, 秦鑫宇. 密度Canopy的增强聚类与深度特征的KNN算法[J]. 计算机科学与探索, 2021, 15(7): 1289-1301.
[5]	徐辉, 祝玉华, 甄彤, 李智慧. 深度神经网络图像语义分割方法综述[J]. 计算机科学与探索, 2021, 15(1): 47-59.
[6]	林阳，初旭，王亚沙，毛维嘉，赵俊峰. 融合自注意力机制的跨模态食谱检索方法[J]. 计算机科学与探索, 2020, 14(9): 1471-1481.
[7]	李俊杰，王茜. 感知相似的图像分类对抗样本生成模型[J]. 计算机科学与探索, 2020, 14(11): 1930-1942.
[8]	张涛，任相赢，刘阳，耿彦章. 基于自编码特征的语音增强声学特征提取[J]. 计算机科学与探索, 2019, 13(8): 1341-1350.
[9]	徐毅，董晴，戴鑫，宋威. ELM优化的深度自编码分类算法[J]. 计算机科学与探索, 2018, 12(5): 820-827.
[10]	王毅，冯小年，钱铁云，朱辉，周静. 基于CNN和LSTM深度网络的伪装用户入侵检测[J]. 计算机科学与探索, 2018, 12(4): 575-585.
[11]	张威，周治平. 融合注意力和动态语义指导的图像描述模型[J]. 计算机科学与探索, 2017, 11(12): 2033-2040.