Review of Question Difficulty Evaluation Approaches

doi:10.3778/j.issn.1673-9418.2108086

Journal of Frontiers of Computer Science and Technology ›› 2022, Vol. 16 ›› Issue (4): 734-759.DOI: 10.3778/j.issn.1673-9418.2108086

• Surveys and Frontiers • Previous Articles Next Articles

Review of Question Difficulty Evaluation Approaches

XU Jia¹^,²^,³^,⁺(), WEI Tingting¹, YU Ge⁴, HUANG Xinyue¹, LYU Pin¹^,²^,³

1. School of Computer Electronics and Information, Guangxi University, Nanning 530004, China
2. Guangxi Key Laboratory of Multimedia Communications and Network Technology, Guangxi University, Nanning 530004, China
3. Guangxi Colleges and University Key Laboratory of Parallel and Distributed Computing, Guangxi University, Nanning 530004, China
4. School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China

Received:2021-08-23 Revised:2021-11-24 Online:2022-04-01 Published:2021-12-01
About author:XU Jia, born in 1984, Ph.D., associate professor, M.S. supervisor, senior member of CCF, member of CCF Database Committee. Her research interests include database theory and technology, educational data analysis and mining, etc.
WEI Tingting, born in 1996, M.S. candidate, student member of CCF. Her research interests include exercise difficulty prediction and personalized recommendation.
YU Ge, born in 1962, Ph.D., professor, Ph.D. supervisor, fellow of CCF. His research interests include database theory and technology, parallel and distributed computing, etc.
HUANG Xinyue, born in 1997, M.S. candidate, student member of CCF. Her research interests include big data in education and data mining.
LYU Pin, born in 1983, Ph.D., associate professor, M.S. supervisor, senior member of CCF, member of CCF Cooperative Computing. His research interests include Internet of things, educational big data analysis, etc.
Supported by:
National Natural Science Foundation of China(62067001);National Natural Science Foundation of China(U1811261);Special Funds for Guangxi BaGui Scholars, the Projects of Higher Education Undergraduate Teaching Reform in Guangxi(2020JGA116);Special Funds for Guangxi BaGui Scholars, the Projects of Higher Education Undergraduate Teaching Reform in Guangxi(2017JGZ103);Innovation Project of Guangxi Graduate Education(JGY2021003);Natural Science Foundation of Guangxi(2019JJA170045)

题目难度评估方法研究综述

许嘉¹^,²^,³^,⁺(), 韦婷婷¹, 于戈⁴, 黄欣悦¹, 吕品¹^,²^,³

1.广西大学计算机与电子信息学院,南宁 530004
2.广西大学广西多媒体通信网络技术重点实验室,南宁 530004
3.广西大学广西高校并行与分布式计算重点实验室,南宁 530004
4.东北大学计算机科学与工程学院,沈阳 110819

通讯作者: + E-mail: xujia@gxu.edu.cn
作者简介:许嘉（1984—）,女,山东荣成人,博士,副教授,硕士生导师,CCF高级会员,CCF数据库专委会委员,主要研究方向为数据库理论与技术、教育数据分析挖掘等。
韦婷婷（1996—）,女,广西桂平人,硕士研究生,CCF学生会员,主要研究方向为习题难度预测、个性化推荐。
于戈（1962—）,男,辽宁大连人,博士,教授,博士生导师,CCF会士,主要研究方向为数据库理论与技术、并行分布式计算等。
黄欣悦（1997—）,女,广西人,硕士研究生,CCF学生会员,主要研究方向为教育大数据、数据挖掘。
吕品（1983—）,男,山东滨州人,博士,副研究员,硕士生导师,CCF高级会员,CCF协同计算专委会委员,主要研究方向为物联网、教育大数据分析等。
基金资助:
国家自然科学基金(62067001);国家自然科学基金(U1811261);“广西八桂学者”专项经费；广西高等教育本科教学改革工程项目(2020JGA116);“广西八桂学者”专项经费；广西高等教育本科教学改革工程项目(2017JGZ103);广西研究生教育创新计划资助项目(JGY2021003);广西自然科学基金(2019JJA170045)

Abstract

Abstract:

Difficulty of a question is not only the key information to ensure the rationality of a test paper and the fairness of a test, but also acts as a critical parameter in intelligent tutoring system (ITS), effectively supporting many intelligent teaching functions, such as intelligent paper forming, automatic question generation, and perso-nalized exercise recommendation. Therefore, question difficulty evaluation has become an important research dire-ction in the field of educational data mining, and has a lot of research work. This paper comprehensively reviews the research progress of question difficulty evaluation in recent ten years, divides the question difficulty into two cate-gories: absolute difficulty and relative difficulty, sorts out and classifies exsiting evaluation approaches of question difficulty, and mainly explains deep learning based approaches for both question absolute difficulty prediction and question relative difficulty prediction. Specifically, important approaches of deep learning based question relative difficulty prediction are experimentally analyzed. Meanwhile, related datasets and evaluation metrics of question difficulty prediction approaches are summarized. Finally, the future research directions of question difficulty eva-luation are prospected.

Key words: question difficulty, difficulty evaluation, machine learning, deep learning, knowledge tracking, cogni-tive diagnosis

摘要：

题目难度是保证试卷合理性及考试公平性的关键信息,也是智能教学系统（ITS）中的关键参数,有效支撑着包括智能组卷、题目自动生成和个性化习题推荐在内的多项智能教学功能。因此,题目难度评估已成为教育数据挖掘领域的一个重要研究方向,拥有大量研究工作。全面回顾了近十年题目难度评估研究领域的研究进展,将题目难度分为题目绝对难度和题目相对难度两类,并对现有的题目难度评估方法进行了整理和分类,其中重点分析了基于深度学习的题目绝对难度预测方法和基于深度学习的题目相对难度预测方法,并对后者包含的重要方法进行了实验分析。同时,对题目难度预测的相关数据集和模型评价指标等进行了总结。最后,对题目难度评估的未来研究方向进行了展望。

关键词: 题目难度, 难度评估, 机器学习, 深度学习, 知识追踪, 认知诊断

CLC Number:

TP391

XU Jia, WEI Tingting, YU Ge, HUANG Xinyue, LYU Pin. Review of Question Difficulty Evaluation Approaches[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(4): 734-759.

许嘉, 韦婷婷, 于戈, 黄欣悦, 吕品. 题目难度评估方法研究综述[J]. 计算机科学与探索, 2022, 16(4): 734-759.

Figures/Tables 21

Fig.1 Classification of approaches for question absolute difficulty evaluation

Fig.2 Classification of approaches for question relative difficulty evaluation

Fig.3 Classification of machine learning based approaches for question absolute difficulty prediction

Table1 Frequently-used machine learning models for question absolute difficulty prediction

模型	文献
回归分析	[21,24-25,45-47,49-52,54-56]
支持向量机	[24-25,42,49-53,57]
决策树	[25,44,47,49,52-53]
随机森林	[24-25,43,49-52]
浅层BP神经网络	[41,53]

Fig.4 Architecture of important deep learning based question absolute difficulty prediction models

Table 2 Comparison of deep learning based question absolute difficulty prediction models

模型/框架简称	年份	使用模型	方法简单描述	优点	局限性和使用场景
TACNN^[58]	2017	CNN	基于CNN提出了一种预测英语考试中阅读理解题难度的方法	引入注意力机制限定阅读材料中不同语句对题目的贡献	只利用了题目文本的语义信息,只适用于题目文本信息较为丰富的阅读题
C-MIDP、 R-MIDP、 H-MIDP^[12]	2019	CNN、LSTM	提出了基于CNN和RNN的数学题目难度预测模型C-MIDP和R-MIDP,以及二者的混合模型H-MIDP	利用了题目文本的语义特征,并且考虑了题目的序列语义和逻辑信息	只利用题目中的文本信息,无法处理题目中的图片等信息,适用于题目文本信息较为丰富的阅读题
TCN-DPN^[23]	2019	LSTM	利用深度学习技术提取中文阅读理解题的特征,特征提取阶段训练模型利用了数据量较大的语料库	无需手动提取文本特征,利用了题目文本的语义信息	只利用了题目文本信息,且只适用于题目文本信息较为丰富的阅读题
DAN^[22]	2019	CNN、 Bi-LSTM	基于文档增强注意的神经网络提出了针对医学考试中选择题的难度预测模型	构建相关的文档数据库用于丰富输入,框架的召回模块和困惑模块能模拟学生的答题行为	只适用于可建立扩展文档数据库的选择题,且无法处理带图片的选择题
BEDP^[2]	2019	Capsule neural network	提出一个融合深度多模态嵌入模型和贝叶斯推理的题目难度预测框架,深度多模态嵌入模型获得题目的统一多模态表示,分类器预测题目绝对难度	能处理包含图片的题目,同时利用题目中的文本特征和图像特征	能处理的题目类型有限,如无法提取编程题代码中的逻辑结构信息,适用于带有图片的题目

Table 3 Hybrid cognitive diagnostic

文献	年份	方法简述
DIRT^[36]	2019	利用深度学习技术获取IRT所需参数,将参数输入IRT模型预测题目对于学生的相对难度
PMF-CD^[8]	2017	将认知诊断模型和矩阵分解技术进行结合
FuzzyCDF^[38]	2018	认知诊断模型结合模糊集理论、教育假设
NeuralCD^[37]	2020	结合神经网络的认知诊断模型

Fig.5 Classification of knowledge tracking approaches

Table 4 Extended models for BKT

文献	年份	主要考虑的因素
文献	年份	学生方面	题目方面	其他方面
KT-Forget、KT-Slip^[72]	2011	遗忘因素、失误因素	—	—
KT-IDEM^[73]	2011	—	题目绝对难度	—
Individualized BKT^[39]	2013	学习率、知识点初始掌握程度	—	—
BKT-ST^[74]	2014	—	题目相似性	—
EEG-KT和EEG-LRKT^[75]	2014	心理状态	—	—
LF-KT^[76]	2014	学生能力	题目绝对难度	结合潜在因子模型
DBN^[77]	2014	—	知识点层次结构	—
KT&IRT^[78]	2014	学生能力	题目绝对难度	结合IRT模型
FAST^[79]	2014	—	—	允许将一般特征集成到该模型
KAT^[80]	2014	学生行为特征	—	—
Spectral BKT^[81]	2015	学习状态	—	—
Affective BKT^[82]	2015	心理状态	—	—
Multi-Grained-BKT和Historical-BKT^[83]	2016	遗忘因素、失误因素	知识点层次结构	—
BKT+FSA^[84]	2016	遗忘、学生能力	—	—
Intervention-BKT^[85]	2016	—	—	不同类型的教学干预
TLS-BKT^[86]	2018	学习状态	—	—
TD-BKT^[87]	2018	—	—	时间差异
MS-BKT^[88]	2020	学习率、学习状态	—	—

Fig.6 Architecture of DKT model

Table 5 Extended models of DKT model

文献	年份	主要改进类型
文献	年份	除学生答题交互序列外的模型输入数据	引入注意力机制	模型其他特性
DKT-FE^[90]	2017	学生答题次数、请求提示的次数、答题时间等	否	—
DKT-DT^[91]	2017	多种异构特征,如学生答题尝试次数、请求提示的次数等	否	—
NKT^[92]	2017	—	否	针对DKT的长期依赖问题提出双层堆叠LSTM
Classifier-based DKT^[93]	2018	异构特征,如学生答题次数、请求提示次数、答题时间等	否	—
EERNN^[94]	2018	题目文本信息	是	—
DKT-DSC^[95]	2018	学生能力	否	—
PDKT-C^[96]	2018	题目和知识点之间的关系、知识点之间的先决关系	否	利用 $Q$ 矩阵表示题目和知识点之间的关系
DKT+^[97]	2018	—	否	在损失函数中引入正则化项
E2E-DKT^[98]	2018	题目和知识点之间的关系	否	—
DHKT^[99]	2019	题目和知识点之间的关系	否	利用 $Q$ 矩阵表示题目和知识点之间的关系
DKT+Forgetting^[100]	2019	学生遗忘行为	否	—
DKTS^[101]	2019	题目之间的相似关系	否	—
KQN^[102]	2019	—	否	基于向量点积的自解释模型
BDKT^[103]	2019	题目知识点	否	综合贝叶斯网络和DKT模型
AKTHE^[104]	2020	题目和题目属性之间的关系	是	利用异构信息网络描述题目和其绝对难度、区分度之间的关系
GIKT^[105]	2020	题目和知识点之间的关系	是	利用图卷积网络学习题目和知识点之间关系的嵌入
DynEmb^[106]	2020	与答题相关的其他信息,如答题时间戳、知识点、题目文本等	否	利用矩阵分解技术获取题目的潜在嵌入
qDKT^[107]	2020	题目相似性	否	将题目差异正则化,作为额外的损失函数,并提出一种初始化嵌入矩阵的新方法
A-DKT^[108]	2020	题目相似性	是	—
EHFKT^[109]	2020	题目的语义、知识点和绝对难度	否	—
TC-MIRT^[110]	2021	学生答题的时间间隔	否	将项目反应理论的参数集成到改进的RNN中,增强模型的可解释性

Table 5 Extended models of DKT model

文献	年份	主要改进类型
文献	年份	除学生答题交互序列外的模型输入数据	引入注意力机制	模型其他特性
DKT-FE^[90]	2017	学生答题次数、请求提示的次数、答题时间等	否	—
DKT-DT^[91]	2017	多种异构特征,如学生答题尝试次数、请求提示的次数等	否	—
NKT^[92]	2017	—	否	针对DKT的长期依赖问题提出双层堆叠LSTM
Classifier-based DKT^[93]	2018	异构特征,如学生答题次数、请求提示次数、答题时间等	否	—
EERNN^[94]	2018	题目文本信息	是	—
DKT-DSC^[95]	2018	学生能力	否	—
PDKT-C^[96]	2018	题目和知识点之间的关系、知识点之间的先决关系	否	利用 $Q$ 矩阵表示题目和知识点之间的关系
DKT+^[97]	2018	—	否	在损失函数中引入正则化项
E2E-DKT^[98]	2018	题目和知识点之间的关系	否	—
DHKT^[99]	2019	题目和知识点之间的关系	否	利用 $Q$ 矩阵表示题目和知识点之间的关系
DKT+Forgetting^[100]	2019	学生遗忘行为	否	—
DKTS^[101]	2019	题目之间的相似关系	否	—
KQN^[102]	2019	—	否	基于向量点积的自解释模型
BDKT^[103]	2019	题目知识点	否	综合贝叶斯网络和DKT模型
AKTHE^[104]	2020	题目和题目属性之间的关系	是	利用异构信息网络描述题目和其绝对难度、区分度之间的关系
GIKT^[105]	2020	题目和知识点之间的关系	是	利用图卷积网络学习题目和知识点之间关系的嵌入
DynEmb^[106]	2020	与答题相关的其他信息,如答题时间戳、知识点、题目文本等	否	利用矩阵分解技术获取题目的潜在嵌入
qDKT^[107]	2020	题目相似性	否	将题目差异正则化,作为额外的损失函数,并提出一种初始化嵌入矩阵的新方法
A-DKT^[108]	2020	题目相似性	是	—
EHFKT^[109]	2020	题目的语义、知识点和绝对难度	否	—
TC-MIRT^[110]	2021	学生答题的时间间隔	否	将项目反应理论的参数集成到改进的RNN中,增强模型的可解释性

Fig.7 Architecture of DKVMN model

Table 6 Extended models of DKVMN model

文献	年份	主要改进类型
文献	年份	除学生答题交互序列外的模型输入数据	模型整合
Colearn^[113]	2018	学生获取答案提示的行为	—
EKT^[114]	2019	题目文本、知识点	整合DKVMN模型和EERNN模型
DSCMN^[115]	2019	学生能力	整合DKVMN模型和DKT-DSC模型,将学生按能力分组进行训练以隐式添加学生学习能力特征
SKVMN^[116]	2019	—	整合DKVMN模型和DKT模型
DKVMN-CA^[117]	2019	考虑了学生和题目方面的多个特征（包括题目文本、知识点、学生学习阶段、答题时间）,并设计了基于课程概念表的存储结构	—
DKVMN-DT^[118]	2019	利用决策树对学生答题行为特征进行预处理,学生答题行为特征和答题交互一起作为模型的输入	—
Deep-IRT^[119]	2019	—	整合DKVMN模型和IRT理论,提高模型的可解释性
LFKT^[40]	2021	考虑影响学生知识遗忘的四个因素：学生重复学习知识点的间隔时间、重复学习知识点的次数、顺序学习间隔时间以及学生对知识点的掌握程度	—
DKVMN-LA^[120]	2021	学生学习能力、答题行为特征	—

Fig.8 Architecture of SAKT model

Table 7 Summary of knowledge tracking models based on Transformers

模型	年份	优点	局限性
SAKT^[121]	2019	能较好地处理数据稀疏性问题,即学生答题交互序列不多的情况	无法指出学生在具体知识点上的掌握程度;未考虑学生答题过程中的遗忘行为;模型的注意力层过浅无法捕捉题目和答题结果之间的复杂关系
SAINT^[122]	2020	相较于SAKT模型,能够较好地捕捉题目和答题结果之间的复杂关系	未考虑知识点间的关系;无法指出学生在具体知识点上的掌握程度;未考虑学生答题过程中的遗忘行为
DKTT^[123]	2020	模型不但能自动识别题目中的潜在知识点,还模拟了学生的遗忘行为	无法处理知识点之间的先决关系对学生答题结果的影响

Fig.9 Architecture of GKT model

Fig.10 Architecture of CKT model

Table 8 Comparison of DKT, DKVMN, SAKT, GKT and CKT model

模型大类	时间	初始模型	表征学生知识状态	有开源代码	优点	局限性
DKT^[89]	2015	RNN/LSTM	单个隐藏向量	是	无需专家标注题目中的知识点,相较于BKT模型更适合应用于在线教育环境	模型输入较简单;使用单个隐藏层表示学生的知识状态,无法指出学生不同知识点上的掌握程度;难以捕捉长依赖关系;不适用于处理稀疏数据
DKVMN^[111]	2017	MANN	每个潜在知识点对应一个知识点状态向量	是	显性维护知识点矩阵和对应的知识状态矩阵,增强了模型的可解释性;外部存储量较大,因此参数较DKT模型更少	模型输入较为简单;过于依赖模型本身的遗忘机制;不适用于处理稀疏数据
SAKT^[121]	2019	Transformers	无	否	能较好地处理数据稀疏性问题;模型适合并行计算,训练速度较短;与RNN相比更能捕获长期依赖关系	无法具体指出学生在某个知识点上的掌握程度;无法处理题目知识点之间先决关系对学生答题结果的影响
GKT^[124]	2019	GNN	每个知识点对应一个知识点状态向量	是	解决了无法对知识点之间的复杂关系进行建模的问题;自动提取知识点之间的关系提高了模型的可解释性	只考虑与题目知识点的邻近节点,并未考虑远程节点的影响
CKT^[125]	2020	CNN	学生知识状态矩阵	是	对学习过程中的个性化进行建模;自动学习有意义的题目嵌入	模型未考虑学生答题过程中的遗忘行为

Table 9 Experimental comparison of important deep knowledge tracking models

模型	AUC	训练时间/min
DKT^[89]	0.804 0	97.02
DKVMN^[111]	0.813 0	240.35
GKT^[124]	0.671 9	487.50
CKT^[125]	0.823 9	31.58

Table 10 Public datasets of student interaction sequences

名称	特点简述	语言	链接
Math	高中生的两次数学考试,包括客观题和主观题	英文	http://staff.ustc.edu.cn/%7Eqiliuql/data/math2015.rar
ASSISTments2009	ASSISTments在线教学系统收集的2009—2010学年的小学数学题的答题记录数据,密度为0.06	英文	https://sites.google.com/site/assistmentsdata/home/assistment2009-2010-data/skill-builder-data-2009-2010
ASSISTments2012	ASSISTments在线教学系统收集的2012—2013学年的小学数学题的答题记录数据	英文	https://sites.google.com/site/assistmentsdata/home/2012-13-school-data-with-affect
ASSISTments2015	ASSISTments在线教学系统收集的2014—2015学年的小学数学题的答题记录数据,密度为0.05	英文	https://sites.google.com/site/assistmentsdata/home/2015-assistments-skill-builder-data
ASSISTment Challenges	来自2017年ASSISTment教育数据挖掘挑战赛的数据,密度为0.81	英文	https://sites.google.com/view/assistmentsdatamining
Synthetic-5	模拟2 000名学生答题的数据集,该数据集中每个学生回答50道题,每道题中包含一个知识点,且数据集中包含题目绝对难度信息	英文	https://github.com/chrispiech/DeepKnowledgeTracing/tree/master/data/synthetic
Statics2011	数据来源某大学的工程静态课程	英文	https://pslcdatashop.web.cmu.edu/DatasetInfo?datasetId=507
KDD Cup 2010	2010年KDD Cup比赛的开源数据集	英文	https://pslcdatashop.web.cmu.edu/KDDCup/downloads.jsp
AICFE-*	该系列数据集共包含8个不同的数据集,分别为语文、数学、英语、物理、化学、生物、历史和地理共8个科目,数据收集时间持续近3年	中文	http://www.bnu-ai.cn/download-unit
EdNet	数据来自英语教学系统Santa,是迄今为止最大的开源交互教育系统数据集	英文	https://github.com/riiid/ednet
Junyi Academy	数据来自中国台湾的均一教育平台,除EdNet外数据量最大的开源数据集	中文	https://pslcdatashop.web.cmu.edu/DatasetInfo?datasetId=1198
Slepemapy.cz	数据来自一个用于练习地理学的在线系统	英文	https://www.fi.muni.cz/adaptivelearning/?a=data
Anonymizeddata	数据来自计算机编程挑战	英文	https://code.org/reasearch
NeurIPS 2020 Education Challenge	Eedi在NeurIPS会议上发起的预测建模挑战赛的数据,题型为选择题	英文	https://diagnosticquestions.com/
Datashop	最大的学习交互数据存储库,包含30个以上数据集,涉及多个学科的学习交互数据,数据时间跨度较大且各数据集的数据特征也不尽相同	英文、中文等	https://pslcdatashop.web.cmu.edu/

Table 10 Public datasets of student interaction sequences

名称	特点简述	语言	链接
Math	高中生的两次数学考试,包括客观题和主观题	英文	http://staff.ustc.edu.cn/%7Eqiliuql/data/math2015.rar
ASSISTments2009	ASSISTments在线教学系统收集的2009—2010学年的小学数学题的答题记录数据,密度为0.06	英文	https://sites.google.com/site/assistmentsdata/home/assistment2009-2010-data/skill-builder-data-2009-2010
ASSISTments2012	ASSISTments在线教学系统收集的2012—2013学年的小学数学题的答题记录数据	英文	https://sites.google.com/site/assistmentsdata/home/2012-13-school-data-with-affect
ASSISTments2015	ASSISTments在线教学系统收集的2014—2015学年的小学数学题的答题记录数据,密度为0.05	英文	https://sites.google.com/site/assistmentsdata/home/2015-assistments-skill-builder-data
ASSISTment Challenges	来自2017年ASSISTment教育数据挖掘挑战赛的数据,密度为0.81	英文	https://sites.google.com/view/assistmentsdatamining
Synthetic-5	模拟2 000名学生答题的数据集,该数据集中每个学生回答50道题,每道题中包含一个知识点,且数据集中包含题目绝对难度信息	英文	https://github.com/chrispiech/DeepKnowledgeTracing/tree/master/data/synthetic
Statics2011	数据来源某大学的工程静态课程	英文	https://pslcdatashop.web.cmu.edu/DatasetInfo?datasetId=507
KDD Cup 2010	2010年KDD Cup比赛的开源数据集	英文	https://pslcdatashop.web.cmu.edu/KDDCup/downloads.jsp
AICFE-*	该系列数据集共包含8个不同的数据集,分别为语文、数学、英语、物理、化学、生物、历史和地理共8个科目,数据收集时间持续近3年	中文	http://www.bnu-ai.cn/download-unit
EdNet	数据来自英语教学系统Santa,是迄今为止最大的开源交互教育系统数据集	英文	https://github.com/riiid/ednet
Junyi Academy	数据来自中国台湾的均一教育平台,除EdNet外数据量最大的开源数据集	中文	https://pslcdatashop.web.cmu.edu/DatasetInfo?datasetId=1198
Slepemapy.cz	数据来自一个用于练习地理学的在线系统	英文	https://www.fi.muni.cz/adaptivelearning/?a=data
Anonymizeddata	数据来自计算机编程挑战	英文	https://code.org/reasearch
NeurIPS 2020 Education Challenge	Eedi在NeurIPS会议上发起的预测建模挑战赛的数据,题型为选择题	英文	https://diagnosticquestions.com/
Datashop	最大的学习交互数据存储库,包含30个以上数据集,涉及多个学科的学习交互数据,数据时间跨度较大且各数据集的数据特征也不尽相同	英文、中文等	https://pslcdatashop.web.cmu.edu/

Table 11 Sources of true difficulty lables of questions

来源	文献
专家评估	[2,20,23,26-28,42,44,51,53-56,59,64]
答题数据	[1,21-24,41-42,45-46,49-50,56,58,62-63]

References 133

[1]	PADÓ U. Question difficulty-how to estimate without nor-ming, how to use for automated grading[C]// Proceedings of the 2017 Workshop on Innovative Use of NLP for Building Educational Applications, Copenhagen, Sep 8, 2017. Strou-dsburg: ACL, 2017: 1-10.
[2]	FANG J S, ZHAO W, JIA D Y. Exercise difficulty prediction in online education systems[C]// Proceedings of the 2019 In-ternational Conference on Data Mining Workshops, Beijing, Nov 8-11, 2019. Piscataway: IEEE, 2019: 311-317.
[3]	HAN G Y, LI X Z. An intelligent test paper generation algo-rithm based on adjustment of overall difficulty degrees[J]. Applied Mechanics & Materials, 2013, 411-414:2879-2882.
[4]	HU Z, XING C. An intelligent test paper generation method based on genetic particle swarm optimization[C]// Procee-dings of the 2016 Web Information Systems and Applica-tions Conference, Wuhan, Sep 23-25, 2016. Piscataway: IEEE, 2016: 188-194.
[5]	VINU E V, SREENIVASA K P. Automated generation of as-sessment tests from domain ontologies[J]. Semantic Web, 2017, 8(6):1023-1047. DOI URL
[6]	KHODEIR N A, ELAZHARY H, WANAS N, et al. Gene-rating story problems via controlled parameters in a web-based intelligent tutoring system[J]. The International Jour-nal of Information and Learning Technology, 2018, 35(3):199-216.
[7]	HUO Y J, WONG D F, NI L M, et al. Knowledge modeling via contextualized representations for LSTM-based perso-nalized exercise recommendation[J]. Information Sciences, 2020, 523:266-278. DOI URL
[8]	朱天宇, 黄振亚, 陈恩红, 等. 基于认知诊断的个性化试题推荐方法[J]. 计算机学报, 2017, 40(1):176-191.
	ZHU T Y, HUANG Z Y, CHEN E H, et al. Cognitive diag-nosis based personalized question recommendation[J]. Chinese Journal of Computers, 2017, 40(1):176-191.
[9]	DEVELLIS R F. Classical test theory[J]. Medical Care, 2006, 44(3):S50-9. DOI URL
[10]	HOLLAND P W, THAYER D T. An alternate definition of the ETS delta scale of item difficulty[J]. ETS Research Re-port Series, 1985(2): i-10.
[11]	SUSANTI Y, TOKUNAGA T, NISHIKAWA H, et al. Cont-rolling item difficulty for automatic vocabulary question generation[J]. Research & Practice in Technology Enhanced Learning, 2017, 12(1):25.
[12]	佟威, 汪飞, 刘淇, 等. 数据驱动的数学试题难度预测[J]. 计算机研究与发展, 2019, 56(5):1007-1019.
	TONG W, WANG F, LIU Q, et al. Data driven prediction for the difficulty of mathematical items[J]. Journal of Com-puter Research and Development, 2019, 56(5):1007-1019.
[13]	ZHU G L, LIU W J, ZHANG S X. Designing personalized learning difficulty for online learners[C]// LNCS 6537: Procee-dings of the International Conference on Web-based Lear-ning, Shanghai, Dec 7-11, 2010. Berlin, Heidelberg: Springer, 2010: 264-275.
[14]	TEUSNER R, HILLE T, HAGEDORN C. Aspects on fin-ding the optimal practical programming exercise for MOOCs[C]// Proceedings of the Frontiers in Education Con-ference, Indianapolis, Oct 18-21, 2017. Washington: IEEE Computer Society, 2017: 1-8.
[15]	GAN W B, SUN Y, PENG X, et al. Modeling learner’s dy-namic knowledge construction procedure and cognitive item difficulty for knowledge tracing[J]. Applied Intelligence, 2020, 50(11):3894-3912. DOI URL
[16]	刘恒宇, 张天成, 武培文, 等. 知识追踪综述[J]. 华东师范大学学报(自然科学版), 2019(5):1-15.
	LIU H Y, ZHANG T C, WU P W, et al. A review of know-ledge tracking[J]. Journal of East China Normal University (Natural Sciences), 2019(5):1-15.
[17]	张暖, 江波. 学习者知识追踪研究进展综述[J]. 计算机科学, 2021, 48(4):213-222.
	ZHANG N, JIANG B. Review progress of learner know-ledge tracing[J]. Computer Science, 2021, 48(4):213-222.
[18]	胡学钢, 刘菲, 卜晨阳. 教育大数据中认知跟踪模型研究进展[J]. 计算机研究与发展, 2020, 57(12):2523-2546.
	HU X G, LIU F, BU C Y. Research advances on knowledge tracing models in educational big data[J]. Journal of Com-puter Research and Development, 2020, 57(12):2523-2546.
[19]	ALKHUZAEY S, GRASSO F, PAYNE T R, et al. A sys-tematic review of data-driven approaches to item difficulty prediction[C]// LNCS 12748: Proceedings of the 22nd International Conference on Artificial Intelligence in Education, Utrecht, Jun 14-18, 2021. Cham: Springer, 2021: 29-41.
[20]	刘铁园, 陈威, 常亮, 等. 基于深度学习的知识追踪研究进展[J]. 计算机研究与发展, 2022, 59(1):81-104.
	LIU T Y, CHEN W, CHANG L, et al. Research advances in the knowledge tracing based on deep learning[J]. Journal of Computer Research and Development, 2022, 59(1):81-104.
[21]	CHOI I, MOON Y. Predicting the difficulty of EFL tests based on corpus linguistic features and expert judgment[J]. Language Assessment Quarterly, 2020, 17(1):18-42. DOI URL
[22]	QIU Z P, WU X, FAN W. Question difficulty prediction for multiple choice problems in medical exams[C]// Proceedings of the 28th ACM International Conference on Information and Knowledge Management, Beijing, Nov 3-7, 2019. New York: ACM, 2019: 139-148.
[23]	LIN L H, CHANG T H, HSU F Y. Automated prediction of item difficulty in reading comprehension using long short-term memory[C]// Proceedings of the 2019 International Conference on Asian Language Processing, Shanghai, Nov 15-17, 2019. Piscataway: IEEE, 2019: 132-135.
[24]	YANEVA V, HA L A, BALDWIN P, et al. Predicting item survival for multiple choice questions in a high-stakes me-dical exam[C]// Proceedings of the 2020 Language Resources and Evaluation Conference, Marseille, May 11-16, 2020. Paris: European Language Resources Association, 2020: 6812-6818.
[25]	BENEDETTO L, CAPPELLI A, TURRIN R, et al. R2DE: a NLP approach to estimating IRT parameters of newly ge-nerated questions[C]// Proceedings of the 10th International Conference on Learning Analytics and Knowledge, Frank-furt, Mar 23-27, 2020. New York: ACM, 2020: 412-421.
[26]	PERIKOS I, GRIVOKOSTOPOULOU F, KOVAS K, et al. Automatic estimation of exercises’ difficulty levels in a tuto-ring system for teaching the conversion of natural language into first-order logic[J]. Expert Systems-The Journal of Know-ledge Engineering, 2016, 33(6):569-580.
[27]	GRIVOKOSTOPOULOU F, PERIKOS I, HATZILYGEROUDIS I. Estimating the difficulty of exercises on search algori-thms using a neuro-fuzzy approach[C]// Proceedings of the 27th International Conference on Tools with Artificial Intel-ligence, Vietri sul Mare, Nov 9-11, 2015. Washington: IEEE Computer Society, 2015: 866-872.
[28]	GRIVOKOSTOPOULOU F, HATZILYGEROUDIS I, PERIKOS I. Teaching assistance and automatic difficulty estimation in converting first order logic to clause form[J]. Artificial Inte-lligence Review, 2014, 42(3):347-367.
[29]	RUSCH T, LOWRY P B, MAIR P, et al. Breaking free from the limitations of classical test theory: developing and mea-suring information systems scales using item response theory[J]. Information & Management, 2016, 54(2):189-203. DOI URL
[30]	CONEJO R, GUZMAN E, PEREZ-DE-LA-CRUZ J L, et al. An empirical study on the quantitative notion of task difficulty[J]. Expert Systems with Applications, 2014, 41(2):594-606. DOI URL
[31]	HERMI A, ACHOUR W. Difficulty, discrimination and cog-nitive level of microbiology exam questions in the faculty of medicine of tunis[J]. La Tunisie Medicale, 2015, 93(8):1-3.
[32]	涂冬波, 蔡艳, 高旭亮, 等. 高级认知诊断[M]. 北京: 北京师范大学出版社, 2019.
	TU D B, CAI Y, GAO X L, et al. Advanced cognitive diagnosis[M]. Beijing: Beijing Normal University Publi-shing Group, 2019.
[33]	黄振亚. 面向个性化学习的数据挖掘方法与应用研究[D]. 合肥: 中国科学技术大学, 2020.
	HUANG Z Y. Data mining techniques and applications for personalized learning[D]. Hefei: University of Science and Technology of China, 2020.
[34]	TORRE D L J. DINA model and parameter estimation: a di-dactic[J]. Journal of Educational and Behavioral Statistics, 2009, 34(1):115-130. DOI URL
[35]	FAN X. Item response theory and classical test theory: an empirical comparison of their item/person statistics[J]. Edu-cational & Psychological Measurement, 1998, 58(3):357-381.
[36]	CHENG S, LIU Q, CHEN E H, et al. DIRT: deep learning enhanced item response theory for cognitive diagnosis[C]// Proceedings of the 2019 International Conference on Infor-mation and Knowledge Management, Beijing, Nov 3-7, 2019. New York: ACM, 2019: 2397-2400.
[37]	WANG F, LIU Q, CHEN E H, et al. Neural cognitive diagnosis for intelligent education systems[C]// Proceedings of the 34th AAAI Conference on Artificial Intelligence, the 32nd Innovative Applications of Artificial Intelligence Con-ference, the 10th AAAI Symposium on Educational Adva-nces in Artificial Intelligence, New York, Feb 7-12, 2020. Menlo Park: AAAI, 2020: 6153-6161.
[38]	LIU Q, WU R Z, CHEN E H, et al. Fuzzy cognitive dia-gnosis for modelling examinee performance[J]. ACM Tran-sactions on Intelligent Systems and Technology, 2018, 9(4):1-26.
[39]	YUDELSON M V, KOEDINGER K R, GORDON G J. In-dividualized Bayesian knowledge tracing models[C]// LNCS 7926: Proceedings of the 16th International Conference on Artificial Intelligence in Education, Memphis, Jul 9-13, 2013. Berlin, Heidelberg: Springer, 2013: 171-180.
[40]	李晓光, 魏思齐, 张昕, 等. LFKT: 学习与遗忘融合的深度知识追踪模型[J]. 软件学报, 2021, 32(3):818-830.
	LI X GZ, WEI S Q, ZHANG X, et al. LFKT: deep know-ledge tracing model with learning and forgetting behavior merging[J]. Journal of Software, 2021, 32(3):818-830.
[41]	付佩宣. 基于人工神经网络的C.TEST阅读理解题目难度的预测研究[J]. 华文教学与研究, 2014(4):71-78.
	FU P X. A prediction study on the difficulty of C.TEST rea-ding understanding question based on artificial neural network[J]. TCSOL Studies, 2014(4):71-78.
[42]	HSU F Y, LEE H M, CHANG T H, et al. Automated esti-mation of item difficulty for multiple-choice tests: an appli-cation of word embedding techniques[J]. Information Pro-cessing & Management, 2018, 54(6):969-984.
[43]	LIU C C, JIN Z, YANG Z, et al. Automatically difficulty grading method of “instruction system” question bank based on knowledge tree[C]// Proceedings of the 2017 Genetic and Evolutionary Computation Conference Companion, Berlin, Jul 15-19, 2017. New York: ACM, 2017: 283-284.
[44]	陈荟慧, 熊杨帆, 蒋滔滔, 等. 基于在线测评系统的编程题目难度研究[J]. 现代计算机, 2018(9):26-30.
	CHEN H H, XIONG Y F, JIANG T T, et al. Research on pro-gramming exercises difficulty based on online measure-ment system[J]. Modern Computer, 2018(9):26-30.
[45]	MASRI Y E, FERRARA S, FOLTZ P W, et al. Predicting item difficulty of science national curriculum tests: the case of key stage 2 assessments[J]. The Curriculum Journal, 2017, 28(1):59-82. DOI URL
[46]	PANDAROVA I, SCHMIDT T, HARTIG J, et al. Predicting the difficulty of exercise items for dynamic difficulty adaptation in adaptive language tutoring[J]. International Journal of Artificial Intelligence in Education, 2019, 29(3):342-367. DOI URL
[47]	SANO M. Automated capturing of psycho-linguistic features in reading assessment text[C]// Proceedings of the Annual Meeting of National Council on Measurement in Education, Chicago, Apr 15-19, 2015.
[48]	SANO M. Improvements in automated capturing of psycho-linguistic features in reading assessment text[C]// Proceedings of the Annual Meeting of National Council on Measurement in Education, Washington, Apr 7-11, 2016.
[49]	LOUKINA A, YOON S Y, SAKANO J, et al. Textual complexity as a predictor of difficulty of listening items in language proficiency tests[C]// Proceedings of the 2016 Inter-national Conference on Computational Linguistics, Osaka, Dec 11-16, 2016. Stroudsburg: ACL, 2016: 3245-3253.
[50]	HA L A, YANEVA V, BALDWIN P, et al. Predicting the dif-ficulty of multiple choice questions in a high-stakes medical exam[C]// Proceedings of the 14th Workshop on Innovative Use of NLP for Building Educational Applications, Florence, Aug 2, 2019. Stroudsburg: ACL, 2019: 11-20.
[51]	DESAI T, MOLDOVAN D I. Towards predicting difficulty of reading comprehension questions[C]// Proceedings of the 32nd International Florida Artificial Intelligence Research Society Conference, Sarasota, May 19-22, 2019. Menlo Park: AAAI, 2019: 8-13.
[52]	BENEDETTO L, CAPPELLI A, TURRIN R. et al. Intro-ducing a framework to assess newly created questions with natural language processing[C]// LNCS 12163: Proceedings of the 21st International Conference on Artificial Intelli-gence in Education, Ifrane, Jul 6-10, 2020. Cham: Springer, 2020: 43-54.
[53]	HUTZLER D, DAVID E, AVIGAL M, et al. Learning methods for rating the difficulty of reading comprehension questions[C]// Proceedings of the 2014 International Conference on Software Science, Technology and Engineering, Ramat Gan, Jun 11-12, 2014. Piscataway: IEEE, 2014: 54-62.
[54]	LUMLEY T, ROUTITSKY A, MENDELOVITS J, et al. A framework for predicting item difficulty in reading tests[R]. Vancouver: American Educational Research Association, 2012.
[55]	SEYLER D, YAHYA M, BERBERICH K. Knowledge ques-tions from knowledge graphs[C]// Proceedings of the 2017 ACM SIGIR International Conference on Theory of Infor-mation Retrieval, Amsterdam, Oct 1-4, 2017. New York: ACM, 2017: 11-18.
[56]	SUSANTI Y, NISHIKAWA H, TOKUNAGA T, et al. Item difficulty analysis of English vocabulary questions[C]// Pro-ceedings of the 8th International Conference on Computer Supported Education, Rome, Apr 21-23, 2016. Setubal: Sci-TePress, 2016: 267-274.
[57]	BEINBORN L, ZESCH T, GURVYCH I. Candidate evaluation strategies for improved difficulty prediction of language tests[C]// Proceedings of the 10th Workshop on Innovative Use of NLP for Building Educational Applications, Denver, Jun 4, 2015. Stroudsburg: ACL, 2015: 1-11.
[58]	HUANG Z Y, LIU Q, CHEN E H, et al. Question difficulty prediction for READING problems in standard tests[C]// Proceedings of the 31st AAAI Conference on Artificial Inte-lligence, San Francisco, Feb 4-9, 2017. Menlo Park: AAAI, 2017: 1352-1359.
[59]	PERIKOS I, GRIVOKOSTOPOULOU F, HATZILYGEROUDIS I, et al. Difficulty estimator for converting natural language into first order logic[C]// Proceedings of the 3rd International Con-ference on Intelligent Decision Technologies, Piraeus, Jul 20-22, 2011. Berlin, Heidelberg: Springer, 2011: 135-144.
[60]	VENUGOPAL V E, KUMAR P S. Difficulty-level modeling of ontology-based factual questions[J]. Semantic Web, 2020, 11(6):1023-1036. DOI URL
[61]	BANERJEE S, RAO N J, RAMANATHAN C. Rubrics for assessment item difficulty in engineering courses[C]// Pro-ceedings of the Frontiers in Education Conference, El Paso, Oct 21-24, 2015. Washington: IEEE Computer Society, 2015: 1-8.
[62]	ELNAFFAR S. Using software metrics to predict the diffi-culty of code writing questions[C]// Proceedings of the 2016 Global Engineering Education Conference, Abu Dhabi, Apr 10-13, 2016. Piscataway: IEEE, 2016: 513-518.
[63]	ARYADOUST V. Predicting item difficulty in a language test with an adaptive neuro fuzzy inference system[C]// Procee-dings of the 2013 Workshop on Hybrid Intelligent Models & Applications, Singapore, Apr 16-19, 2013. Piscataway: IEEE, 2013: 43-50.
[64]	GRIVOKOSTOPOULOU F, PERIKOS I, HATZILYGEROUDIS I. Difficulty estimation of exercises on tree-based search al-gorithms using neuro-fuzzy and neuro-symbolic approaches[C]// Proceedings of the 5th International Workshop CIMA-2015, Vietri sul Mare, Nov 9-11, 2015. Cham: Springer, 2015: 75-91.
[65]	KOREN Y, BELL R, VOLINSKY C. Matrix factorization techniques for recommender systems[J]. Computer, 2009, 42(8):30-37.
[66]	CHEN Y Y, LIU Q, HUANG Z Y, et al. Tracking know-ledge proficiency of students with educational priors[C]// Proceedings of the 2017 Conference on Information and Know-ledge Management, Singapore, Nov 6-10, 2017. New York: ACM, 2017: 989-998.
[67]	SALAKHUTDINOV R, MNIH A. Probabilistic matrix fac-torization[C]// Proceedings of the Annual Conference on Neural Information Processing Systems, Vancouver, Dec 3-6, 2007. New York: Curran Associates, 2007: 1257-1264.
[68]	CORBETT A T, ANDERSON J R. Knowledge tracing: modeling the acquisition of procedural knowledge[J]. User Modeling and User-Adapted Interaction, 1995, 4(4):253-278. DOI URL
[69]	BHATT S P, ZHAO J J, THILLE C. Evaluating Bayesian knowledge tracing for estimating learner proficiency and guiding learner behavior[C]// Proceedings of the 2020 Con-ference on Learning @Scale, Aug 12-14, 2020. New York: ACM, 2020: 357-360.
[70]	DOROUDI S, BRUNSKILL E. The misidentified identifia-bility problem of Bayesian knowledge tracing[C]// Procee-dings of the 10th International Conference on Educational Data Mining, Wuhan, Jun 25-28, 2017. Worcester: IEDMS, 2017: 143-149.
[71]	DEMPSTER A P, LAIRD N M, RUBIN D B. Maximum likelihood from incomplete data via the EM algorithm[J]. Journal of the Royal Statistical Society, 1977, 39(1):1-38.
[72]	QIU Y M, QI Y M, LU H Y, et al. Does time matter? Mode-ling the effect of time with Bayesian knowledge tracing[C]// Proceedings of the 2011 International Conference on Educa-tional Data Mining, Eindhoven, Jul 6-8, 2011. Worcester: IEDMS, 2011: 139-148.
[73]	PARDOS Z A, HEFFERNAN N T. KT-IDEM: introducing item difficulty to the knowledge tracing model[C]// LNCS 6787: Proceedings of the 19th Conference on User Modeling, Adaptation, and Personalization, Girona, Jul 11-15, 2011. Berlin, Heidelberg: Springer, 2011: 243-254.
[74]	HAWKINS W J, HEFFERNAN N T. Using similarity to the previous problem to improve Bayesian knowledge tracing[C]// Proceedings of the 2014 International Conference on Educational Data Mining, London, Jul 4-7, 2014. Worcester: IEDMS, 2014: 1-5.
[75]	XU Y B, CHANG K M, YUAN Y R, et al. Using EEG in knowledge tracing[C]// Proceedings of the 2014 International Conference on Educational Data Mining, London, Jul 4-7, 2014. Worcester: IEDMS, 2014: 361-362.
[76]	KHAJAH M, WING R W, LINDSEY R V, et al. Integrating latent-factor and knowledge-tracing models to predict indi-vidual differences in learning[C]// Proceedings of the 2014 International Conference on Educational Data Mining, Lon-don, Jul 4-7, 2014. Worcester: IEDMS, 2014: 99-106.
[77]	KÄSER T, KLINGLER S, SCHWING A G, et al. Beyond knowledge tracing: modeling skill topologies with Bayesian networks[C]// LNCS 8474: Proceedings of the 12th International Conference on Intelligent Tutoring Systems, Honolulu, Jun 5-9, 2014. Cham: Springer, 2014: 188-198.
[78]	KHAJAH M, HUANG Y, GONZALEZ-BRENES J P, et al. Integrating knowledge tracing and item response theory: a tale of two frameworks[C]// Proceedings of the 22nd Confe-rence on User Modeling, Adaptation, and Personalization, Aalborg, Jul 7-11, 2014. Cham: Springer, 2014: 7-15.
[79]	HUANG Y, GONZALEZ-BRENES J P, BRUSILOVSKY P. General features in knowledge tracing to model multiple subskills, temporal item response theory, and expert knowl-edge[C]// Proceedings of the 7th International Conference on Educational Data Mining, London, Jul 4-7, 2014. Worcester: IEDMS, 2014: 84-91.
[80]	SCHULTZ S E, ARROYO I. Tracing knowledge and enga-gement in parallel in an intelligent tutoring system[C]// Pro-ceedings of the 2014 International Conference on Educational Data Mining, London, Jul 4-7, 2014. Worcester: IEDMS, 2014: 312-315.
[81]	FALAKMASIR M H, YUDELSON M, RITTER S, et al. Spectral Bayesian knowledge tracing[C]// Proceedings of the 2015 International Conference on Educational Data Mining, Madrid, Jun 26-29, 2015. Worcester: IEDMS, 2015: 360-363.
[82]	SPAULDING S, BREAZEAL C. Affect and inference in Bayesian knowledge tracing with a robot tutor[C]// Procee-dings of the Annual International Conference on Human-Robot Interaction, Portland, Mar 2-5, 2015. New York: ACM, 2015: 219-220.
[83]	WANG Z, ZHU J L, LI X, et al. Structured knowledge tracing models for student assessment on coursera[C]// Pro-ceedings of the 2016 Conference on Learning@Scale, Edin-burgh, Apr 25-26, 2016. New York: ACM, 2016: 209-212.
[84]	KHAJAH M, LINDSEY R V, MOZER M C. How deep is knowledge tracing?[C]// Proceedings of the 2016 International Conference on Educational Data Mining, Raleigh, Jun 29-Jul 2, 2016. Worcester: IEDMS, 2016: 94-101.
[85]	LIN C, CHI M. Intervention-BKT: incorporating instructional interventions into Bayesian knowledge tracing[C]// LNCS 9684: Proceedings of the 13th International Conference on Intelligent Tutoring Systems, Zagreb, Jun 7-10, 2016. Cham: Springer, 2016: 208-218.
[86]	ZHANG K, YAO Y Y. A three learning states Bayesian know-ledge tracing model[J]. Knowledge-Based Systems, 2018, 148:189-201. DOI URL
[87]	ZHU J H, ZANG Y C, QIU H, et al. Integrating temporal information into knowledge tracing: a temporal difference approach[J]. IEEE Access, 2018, 6:27302-27312. DOI URL
[88]	AGARWAL D, BAKER R S. Dynamic knowledge tracing through data driven recency weights[C]// Proceedings of the 2020 International Conference on Educational Data Mining, Jul 10-13, 2020. Worcester: IEDMS, 2020: 1-5.
[89]	PIECH C, SPENCER J, HUANG J, et al. Deep knowledge tracing[C]// Proceedings of the Annual Conference on Neural Information Processing Systems, Montreal, Dec 7-12, 2015. Red Hook: Curran Associates, 2015: 505-513.
[90]	ZHANG L, XING X L, ZHAO S Y, et al. Incorporating rich features into deep knowledge tracing[C]// Proceedings of the 2017 Conference on Learning@Scale, Cambridge, Apr 20-21, 2017. New York: ACM, 2017: 169-172.
[91]	CHEUNG L P, YANG H Q. Heterogeneous features integra-tion in deep knowledge tracing[C]// LNCS 10635: Proceedings of the 24th International Conference on Neural Information Processing, Guangzhou, Nov 14-18, 2017. Cham: Springer, 2017: 653-662.
[92]	SHA L, HONG P Y. Neural knowledge tracing[C]// LNCS 10512: Proceedings of the 1st International Conference on Brain Function Assessment in Learning, Patras, Sep 24-25, 2017. Cham: Springer, 2017: 108-117.
[93]	YANG H Q, CHEUNG L P. Implicit heterogeneous features embedding in deep knowledge tracing[J]. Cognitive Com-putation, 2018, 10(1):3-14.
[94]	SU Y, LIU Q W, LIU Q, et al. Exercise-enhanced sequential modeling for student performance prediction[C]// Procee-dings of the 32nd AAAI Conference on Artificial Intelli-gence, the 30th Innovative Applications of Artificial Intelli-gence, and the 8th AAAI Symposium on Educational Adva-nces in Artificial Intelligence, New Orleans, Feb 2-7, 2018. Menlo Park: AAAI, 2018: 2435-2443.
[95]	MINN S, YU Y, DESMARAIS M C, et al. Deep knowledge tracing and dynamic student classification for knowledge tracing[C]// Proceedings of the 2018 International Confe-rence on Data Mining, Singapore, Nov 17-20, 2018. Pisca-taway: IEEE, 2018: 1182-1187.
[96]	CHEN P H, YU L, ZHENG V W, et al. Prerequisite-driven deep knowledge tracing[C]// Proceedings of the 2018 Interna-tional Conference on Data Mining, Singapore, Nov 17-20, 2018. Washington: IEEE Computer Society, 2018: 39-48.
[97]	YEUNG C K, YEUNG D Y. Addressing two problems in deep knowledge tracing via prediction-consistent regulari-zation[C]// Proceedings of the 2018 Conference on Lear-ning@Scale, London, Jun 26-28, 2018. New York: ACM, 2018: 1-10.
[98]	NAKAGAWA H, IWASAWA Y, MATSUO Y. End-to-end deep knowledge tracing by learning binary question-embedding[C]// Proceedings of the 2018 IEEE International Confe-rence on Data Mining Workshops, Singapore, Nov 17-20, 2018. Piscataway: IEEE, 2018: 334-342.
[99]	WANG T Q, MA F L, GAO J. Deep hierarchical knowledge tracing[C]// Proceedings of the 12th International Confe-rence on Educational Data Mining, Montreal, Jul 2-5, 2019. Worcester: IEDMS, 2019: 1-4.
[100]	NAGATANI K, ZHANG Q, SATO M, et al. Augmenting knowledge tracing by considering forgetting behavior[C]// Proceedings of the 2019 International Conference on World Wide Web, San Francisco, May 13-17, 2019. New York: ACM, 2019: 3101-3107.
[101]	WANG Z W, FENG X Q, TANG J L, et al. Deep know-ledge tracing with side information[C]// LNCS 11626: Pro-ceedings of the 20th International Conference on Artificial Intelligence in Education, Chicago, Jun 25-29, 2019. Cham: Springer, 2019: 303-308.
[102]	LEE J, YEUNG D Y. Knowledge query network for know-ledge tracing: how knowledge interacts with skills[C]// Pro-ceedings of the 2019 International Conference on Learning Analytics & Knowledge, Tempe, Mar 4-8, 2019. New York: ACM, 2019: 491-500.
[103]	LI D H, JIA Y M, ZHOU J, et al. Deep knowledge tracing based on Bayesian neural network[C]// Proceedings of the 2019 International Conference on Intelligent and Interac-tive Systems and Applications, Bangkok, Jun 28-30, 2019. Cham: Springer, 2019: 29-37.
[104]	ZHANG N, DU Y, DENG K, et al. Attention-based know-ledge tracing with heterogeneous information network em-bedding[C]// LNCS 12274: Proceedings of the 13th Inter-national Conference on Knowledge Science, Engineering and Management, Hangzhou, Aug 28-30, 2020. Cham: Springer, 2020: 95-103.
[105]	YANG Y, SHEN J, QU Y, et al. GIKT: a graph-based inte-raction model for knowledge tracing[C]// LNCS 12457: Pro-ceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases, Ghent, Sep 14-18, 2020. Cham: Springer, 2020: 299-315.
[106]	XU L B, DAVENPORT M A. Dynamic knowledge embed-ding and tracing[C]// Proceedings of the 2020 International Conference on Educational Data Mining, Jul 10-13, 2020. Worcester: IEDMS, 2020: 1-7.
[107]	SONKAR S, LAN A S, WATERS A E, et al. qDKT: question-centric deep knowledge tracing[C]// Proceedings of the 2020 International Conference on Educational Data Mining, Jul 10-13, 2020. Worcester: IEDMS, 2020: 1-6.
[108]	LIU D, DAI H H, ZHANG Y P, et al. Deep knowledge trac-king based on attention mechanism for student perfor-mance prediction[C]// Proceedings of the 2020 International Conference on Computer Science and Educational Infor-matization, Xinxiang, Jun 12-14, 2020. Piscataway: IEEE, 2020: 95-98.
[109]	TONG H S, ZHOU Y, WANG Z. Exercise hierarchical fea-ture enhanced knowledge tracing[C]// LNCS 12164: Procee-dings of the 2020 International Conference on Artificial Intelligence in Education, Ifrane, Jul 6-10, 2020. Cham: Springer, 2020: 324-328.
[110]	SU Y, CHENG Z Y, LUO P F, et al. Time-and-concept en-hanced deep multidimensional item response theory for in-terpretable knowledge tracing[J]. Knowledge-Based Systems, 2021, 218:106819. DOI URL
[111]	ZHANG J N, SHI X J, KING I, et al. Dynamic key-value memory networks for knowledge tracing[C]// Proceedings of the 2017 International Conference on World Wide Web, Perth, Apr 3-7, 2017. New York: ACM, 2017: 765-774.
[112]	GRAVES A, WAYNE G, REYNOLDS M, et al. Hybrid computing using a neural network with dynamic external memory[J]. Nature, 2016, 538(7626):471-476. DOI URL
[113]	RITWICK C, HARVINEET S, PRADEEP D, et al. Mode-ling hint-taking behavior and knowledge state of students with multi-task learning[C]// Proceedings of the 2018 Inter-national Conference on Educational Data Mining, Buffal, Jul 15-18, 2018. Worcester: IEDMS, 2018: 1-11.
[114]	LIU Q, HUANG Z, YIN Y, et al. EKT: exercise-aware know-ledge tracing for student performance prediction[J]. IEEE Transactions on Knowledge and Data Engineering, 2019, 33(1):100-115. DOI URL
[115]	MINN S, DESMARAIS M, ZHU F, et al. Dynamic student classification on memory networks for knowledge tracing[C]// LNCS 11440: Proceedings of the 2019 Pacific-Asia Conference on Knowledge Discovery and Data Mining, Macau, China, Apr 14-17, 2019. Cham: Springer, 2019: 163-174.
[116]	ABDELRAHMAN G, WANG Q. Knowledge tracing with sequential key-value memory networks[C]// Proceedings of the 2019 International Conference on Research and Deve-lopment in Information Retrieval, Paris, Jul 21-25, 2019. New York: ACM, 2019: 175-184.
[117]	AI F, CHEN Y, GUO Y. Concept-aware deep knowledge tracing and exercise recommendation in an online learning system[C]// Proceedings of the 12th International Conference on Educational Data Mining, Montréal, Jul 2-5, 2019. Wo-rcester: IEDMS, 2019: 1-6.
[118]	SUN X, ZHAO X, MA Y, et al. Muti-behavior features based knowledge tracking using decision tree improved DKVMN[C]// Proceedings of the 2019 ACM Turing Celebration Con-ference, Chengdu, May 17-19, 2019. New York: ACM, 2019: 72.
[119]	YEUNG C K. Deep-IRT: make deep learning based know-ledge tracing explainable using item response theory[C]// Proceedings of the 12th International Conference on Edu-cational Data Mining, Montréal, Jul 2-5, 2019. Worcester: IEDMS, 2019: 1-10.
[120]	SUN X, ZHAO X, LI B, et al. Dynamic key-value memory networks with rich features for knowledge tracing[J]. IEEE Transactions on Cybernetics, 2021: 1-7.
[121]	PANDEY S, KARYPIS G. A self-attentive model for know-ledge tracing[C]// Proceedings of the 2019 International Con-ference on Educational Data Mining, Montréal, Jul 2-5, 2019. Worcester: IEDMS, 2019: 1-6.
[122]	CHOI Y, LEE Y, CHO J, et al. Towards an appropriate query, key, and value computation for knowledge tracing[C]// Pro-ceedings of the 2020 Conference on Learning@Scale, Aug 12-14, 2020. New York: ACM, 2020: 341-344.
[123]	PU S, YUDELSON M, OU L, et al. Deep knowledge tra-cing with transformers[C]// LNCS 12164: Proceedings of the International Conference on Artificial Intelligence in Educa-tion, Ifrane, Jul 6-10, 2020. Cham: Springer, 2020: 252-256.
[124]	NAKAGAWA H, IWASAWA Y, MATSUO Y. Graph-based knowledge tracing: modeling student proficiency using graph neural network[C]// Proceedings of the 2019 ACM Interna-tional Conference on Web Intelligence, Thessaloniki, Oct 14-17, 2019. New York: ACM, 2019: 156-163.
[125]	SHEN S H, LIU Q, CHEN E H, et al. Convolutional know-ledge tracing: modeling individualization in student learning process[C]// Proceedings of the 2020 International Con-ference on Research and Development in Information Ret-rieval, Xi’an, Jul 25-30, 2020. New York: ACM, 2020: 1857-1860.
[126]	WU J Z, HUANG Z Y, LIU Q, et al. Federated deep know-ledge tracing[C]// Proceedings of the 2021 International Con-ference on Web Search and Data Mining, Israel, Mar 8-12, 2021. New York: ACM, 2021: 662-670.
[127]	LIU S, ZOU R, SUN J W, et al. A hierarchical memory net-work for knowledge tracing[J]. Expert Systems with Appli-cations, 2021, 177:114935.
[128]	VIE J J. Deep factorization machines for knowledge tracing[C]// Proceedings of the 2018 Workshop on Innovative Use of NLP for Building Educational Applications@NAACL-HLT, New Orleans, Jun 5, 2018. New York: ACM, 2018: 370-373.
[129]	GUO H F, TANG R M, YE Y M, et al. DeepFM: a factori-zation-machine based neural network for CTR prediction[C]// Proceedings of the 2017 International Joint Confe-rence on Artificial Intelligence, Melbourne, Aug 19-25, 2017. Palo Alto: AAAI, 2017: 1725-1731.
[130]	THAI-NGHE N, HORVATH T, SCHMIDT-THIEME L. Fac-torization models for forecasting student performance[C]// Proceedings of the 2011 International Conference on Edu-cational Data Mining, Eindhoven, Jul 6-8, 2011. Worcester: IEDMS, 2011: 11-20.
[131]	LIU Y F, YANG Y, CHEN X Y, et al. Improving know-ledge tracing via pretraining question embeddings[C]// Proceedings of the 29th International Joint Conference on Artificial Intelligence, Yokohama, Jan 7-15, 2021. Palo Alto: AAAI, 2021: 1577-1583.
[132]	HUANG Z Y, LIU Q, CHEN E H, et al. Learning or forget-ting? A dynamic approach for tracking the knowledge profi-ciency of students[J]. ACM Transactions on Information Systems, 2020, 38(2):1-33.
[133]	VIE J J, KASHIMA H. Knowledge tracing machines: facto-rization machines for knowledge tracing[C]// Proceedings of the 2019 AAAI Conference on Artificial Intelligence, Honolulu, Jan 27-Feb 1, 2019. Menlo Park: AAAI, 2019: 750-757.

Review of Question Difficulty Evaluation Approaches

题目难度评估方法研究综述

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 21

References 133

Related Articles 15

Recommended Articles

Metrics

[1]	AN Fengping, LI Xiaowei, CAO Xiang. Medical Image Classification Algorithm Based on Weight Initialization-Sliding Window CNN [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(8): 1885-1897.
[2]	ZENG Fanzhi, XU Luqian, ZHOU Yan, ZHOU Yuexia, LIAO Junwei. Review of Knowledge Tracing Model for Intelligent Education [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(8): 1742-1763.
[3]	LIU Yi, LI Mengmeng, ZHENG Qibin, QIN Wei, REN Xiaoguang. Survey on Video Object Tracking Algorithms [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(7): 1504-1515.
[4]	ZHAO Xiaoming, YANG Yijiao, ZHANG Shiqing. Survey of Deep Learning Based Multimodal Emotion Recognition [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(7): 1479-1503.
[5]	XIA Hongbin, XIAO Yifei, LIU Yuan. Long Text Generation Adversarial Network Model with Self-Attention Mechanism [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(7): 1603-1610.
[6]	XIE Xintong, HU Yueyang, LIU Xuanzhe, ZHAO Yaoshuai, JIANG Hai’ou. Rumor Detection Based on Representative User Characteristics Learning Through Propagation [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(6): 1334-1342.
[7]	SUN Fangwei, LI Chengyang, XIE Yongqiang, LI Zhongbo, YANG Caidong, QI Jin. Review of Deep Learning Applied to Occluded Object Detection [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(6): 1243-1259.
[8]	LIU Yafen, ZHENG Yifeng, JIANG Lingyi, LI Guohe, ZHANG Wenjie. Survey on Pseudo-Labeling Methods in Deep Semi-supervised Learning [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(6): 1279-1290.
[9]	CHENG Weiyue, ZHANG Xueqin, LIN Kezheng, LI Ao. Deep Convolutional Neural Network Algorithm Fusing Global and Local Features [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(5): 1146-1154.
[10]	ZHONG Mengyuan, JIANG Lin. Review of Super-Resolution Image Reconstruction Algorithms [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(5): 972-990.
[11]	PEI Lishen, ZHAO Xuezhuan. Survey of Collective Activity Recognition Based on Deep Learning [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(4): 775-790.
[12]	ZHU Weijie, CHEN Ying. Micro-expression Recognition Convolutional Network for Dual-stream Temporal-Domain Information Interaction [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(4): 950-958.
[13]	WU Kaijun, HUANG Tao, WANG Dicong, BAI Chenshuai, TAO Xiaomiao. Research Progress of Video Anomaly Detection Technology [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(3): 529-540.
[14]	LIU Liping, SUN Jian, GAO Shiyan. Overview of Blind Deblurring Methods for Single Image [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(3): 552-564.
[15]	LIU Ying, GUO Yingying, FANG Jie, FAN Jiulun, HAO Yu, LIU Jiming. Survey of Research on Deep Learning Image-Text Cross-Modal Retrieval [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(3): 489-511.