计算机科学与探索 ›› 2011, Vol. 5 ›› Issue (8): 730-739.

• 学术研究 • 上一篇    下一篇

MapReduce框架下基于字符串波形的实体识别方法

刘永楠, 王宏志, 高 宏   

  1. 哈尔滨工业大学 计算机科学与技术学院, 哈尔滨 150001
  • 收稿日期:1900-01-01 修回日期:1900-01-01 出版日期:2011-08-01 发布日期:2011-08-01

Entity Resolution Method Based on Wave of Strings Using MapReduce

LIU Yongnan, WANG Hongzhi, GAO Hong

  

  1. School of Computer Science and Technology, Harbin Institute of Technology, Harbin 150001, China
  • Received:1900-01-01 Revised:1900-01-01 Online:2011-08-01 Published:2011-08-01

摘要: 在云计算平台下需要读取大量数据进行分析, 数据中大量指代同一实体的重复数据给数据的分析和处理带来了困难。基于数据记录间的相似度进行聚类分析是目前实体识别的主要方法之一, 但其耗时较长,而且不适用于云计算环境。给出了一种能够很好地利用云计算特点的基于字符串波形的实体识别方法。该方法首先统计字符频率, 按照字符频率的大小生成字符串的波形, 再利用基于波形的过滤性质加快相似度的计算, 进行基于相似度的聚类。理论分析和通过真实数据得出的实验结果都表明了这种方法的正确性和有效性。

关键词: 云计算, MapReduce, 字符串波形, 实体识别

Abstract: Large quantities of records need to be read and analyzed in cloud computing, many records referring to the same entity bring challenges for data processing and analysis. Clustering based on records similarity is one of the most commonly used methods in entity resolution. But existing methods of computing records similarity often cost much time and are not suitable for cloud computing. This paper discusses that it’s necessary to use wave of strings, which can achieve a high performance by using features of cloud computing, to compute records similarity, and provides a method based on wave of strings of entity resolution. The method calculates frequencies of characters, generates waves based on frequencies of characters, and computes records similarity quickly by filtering method based on waves to get clusters based on records similarity. Theoretical analysis and experimental results from real data show that the method is correct and effective.

Key words: cloud computing, MapReduce, wave of strings, entity resolution