带有可信度标记的增量式数据修复方法研究

doi:10.3778/j.issn.1673-9418.2006079

摘要/Abstract

摘要：

大数据时代，数据蕴含着巨大价值，成为当今信息社会的重要战略资源。然而，在对数据进行加工、处理的过程中，产生了大量不一致数据，对企业决策造成了不可预知的恶劣影响。现有的工作主要基于函数依赖研究数据修复技术，已有的修复方法分为三类：前两类需要企业提供Master数据库或给定元组的可信度值，然而在实际应用中，这样的条件未必能满足；而第三类基于最少删除原则的修复方法又会造成信息的丢失。同时，当函数依赖[X→Y]存在冲突时，现有的方法仅支持修改Y属性值。针对以上不足，在没有给定元组可信度的情形下，提出了带有可信度标记的增量式数据修复方法，方法分为两部分：第一部分为通过分析操作日志和知识规则，自动生成单元格的可信度标记；第二部分包含增量式修复策略，依据可信度标记值，确定修复[X]或[Y]属性值，同时结合条件概率来选取目标值进行修复。实验结果表明，所提的修复方法具有较高的可靠性和扩展性。

关键词: 可信度标记（CVT）, 增量式数据修复, 函数依赖, 操作日志

Abstract:

In the era of big data, data contain great value and become important strategic resource in today??s information society. However, a large number of inconsistent data occur during the process of data update and management, which causes unpredictable side effects for enterprises. There are three repair methods based on functional dependencies. The first two methods strongly rely on the Master data or confidence value of given tuples provided by enterprises, which are hard to fulfill in real application. And the third kind of repair method based on the minimal deletion principle will cause the loss of information. Moreover, when solving the conflicts of [X→Y], existing methods only support modifying Y attribute. In view of the shortcomings mentioned above, with the situation of missing tuple confidence, this paper proposes an increased data repair with confidence value token, which can be divided into two parts: the first part is to generate confidence value token automatically by analyzing operator log and knowledge rules, and the second part includes an increased repair strategy which can determine the repair of X or Y attributes according to the confidence value token. Meanwhile, the target value is chosen to repair dirty data with the combination of conditional probability. Experimental results show that the proposed method has high reliability and scalability.

Key words: confidence value token (CVT), increased data repair, functional dependencies, operator log

黄慧, 李海林. 带有可信度标记的增量式数据修复方法研究[J]. 计算机科学与探索, 2021, 15(10): 1900-1911.

HUANG Hui, LI Hailin. Research on Increased Data Repair with Confidence Value Token[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(10): 1900-1911.

参考文献

[1] WAHYUDI A, KUK G, JANSSEN M. A process pattern model for tackling and improving big data quality[J]. Information Systems Frontiers, 2018, 20(3): 457-469.
[2] REDMAN T C. The impact of poor data quality on the typical enterprise[J]. Communications of the ACM, 1998, 41(2): 79-82.
[3] SWARTZ N. Gartner warns firms of “dirty data”[J]. Information Management, 2007, 41(3): 6.
[4] Institute of Medicine. To err is human: building a safer health system[J]. Frontiers of Health Services Management, 2006, 18(6): 453-454.
[5] CUNHA C R. Enterprise information portals[J]. Electronic Library, 2002, 18(5): 354-362.
[6] LI W B, LI Z H, JIANG T. Violations detection of multiple functional dependencies in distributed big data[J]. Chinese Journal of Computers, 2017, 40(1): 146-162.
李卫榜, 李战怀, 姜涛. 分布式大数据多函数依赖冲突检测[J]. 计算机学报, 2017, 40(1): 146-162.
[7] ROBLOT T, LINK S. Cardinality constraints and functional dependencies over possibilistic data[J]. Data & Knowledge Engineering, 2018, 117: 339-358.
[8] CHOMICKI J, MARCINKOWSKI J. Minimal-change integrity maintenance using tuple deletions[J]. Information and Computation, 2005, 197(1/2): 90-121.
[9] FAN W, LI J, MA S, et al. Towards certain fixes with editing rules and master data[J]. VLDB Journal, 2012, 21(2): 213-238.
[10] BOHANNON P, FLASTER M, FAN W F, et al. A cost-based model and effective heuristic for repairing constraints by value modification[C]//Proceedings of the 2005 ACM SIGMOD International Conference on Management of Data, Baltimore, Jun 14-16, 2005. New York: ACM, 2005: 143-154.
[11] FAN W F, MA S, TANG N, et al. Interaction between record matching and data repairing[J]. Journal of Data and Information Quality, 2014, 4(4): 16.
[12] BESKALES G, ILYAS I F, GOLAB L. Sampling the repairs of functional dependency violations under hard constraints[J]. Proceedings of the VLDB Endowment, 2010, 3(1): 197-207.
[13] JIN C Q, LIU H P, ZHOU A Y. Functional dependency and conditional constraint based data repair[J]. Journal of Software, 2016, 27(7): 1671-1684.
金澈清, 刘辉平, 周傲英. 基于函数依赖与条件约束的数据修复方法[J]. 软件学报, 2016, 27(7): 1671-1684.
[14] DUAN X L, GUO B, SHEN Y, et al. Data repair algorithm based on currency rules[J]. Journal of Software, 2019, 30(3): 589-603.
段旭良, 郭兵, 沈艳, 等. 基于时效规则的数据修复方法[J]. 软件学报, 2019, 30(3): 589-603.
[15] DU Y F, SHEN D R, NIE T Z, et al. A cleaning method for consistency and currency in related data[J]. Chinese Journal of Computers, 2017, 40(1): 92-106.
杜岳峰, 申德荣, 聂铁铮, 等. 基于关联数据的一致性和时效性清洗方法[J]. 计算机学报, 2017, 40(1): 92-106.
[16] GOLAB L, KARLOFF H, KORN F, et al. Sequential dependencies[J]. Proceedings of the VLDB Endowment, 2009, 2(1): 574-585.
[17] FAN W F, LI J Z, TANG N, et al. Incremental detection of inconsistencies in distributed data[J]. IEEE Transactions on Knowledge and Data Engineering, 2014, 26(6): 1367-1383.
[18] WU A H, WANG X S, TAN Z J. A cost-based heuristic algorithm for repairing inconsistent XML document[J]. Journal of Software, 2009, 20(4): 918-929.
吴爱华, 王先胜, 谈子敬. 基于代价模型的不一致XML数据修复启发式计算[J]. 软件学报, 2009, 20(4): 918-929.