关键词: 数据一致性, MapReduce, 条件函数依赖, 数据质量

Abstract: With the popularity of the Internet applications in modern society, there comes the problem of increasing poor quality data. This paper investigates inconsistency problem in data quality, designs and realizes an inconsistent data detection and reparation algorithm based on Hadoop. By using the conditional functional dependency (CFD) rules in the data dependency theory, inconsistent data can be detected according to the given rules, and reparation scheme is proposed for the inconsistent data, the final reparation dataset, whose deterministic probability has been calculated, satisfies the consistent requirement. At last, this paper proves that the algorithm performs better than those on a single computer through experiments and the runtime grows linearly when the rules are not large.

Key words: data inconsistency, MapReduce, conditional functional dependency, data quality