计算机科学与探索 ›› 2013, Vol. 7 ›› Issue (10): 924-932.DOI: 10.3778/j.issn.1673-9418.1305047

• 学术研究 • 上一篇    下一篇

Pass-Join-K:多分段匹配的相似性连接算法

余海洋1,林  琛1,2,陈  珂3,江  弋1,邹  权1,2+   

  1. 1. 厦门大学 信息科学与技术学院,福建 厦门 361005
    2. 厦门大学 深圳研究院,广东 深圳 518057
    3. 广东石油化工学院 计算机科学与技术系,广东 茂名 525000
  • 出版日期:2013-10-01 发布日期:2013-09-30

Pass-Join-K: Similarity Join Method Based on Multi-Match Partition

YU Haiyang1, LIN Chen1,2, CHEN Ke3, JIANG Yi1, ZOU Quan1,2+   

  1. 1. School of Information Science and Technology, Xiamen University, Xiamen, Fujian 361005, China
    2. Shenzhen Research Institute of Xiamen University, Shenzhen, Guangdong 518057, China
    3. Department of Computer Science and Technology, Guangdong University of Petrochemical Technology, Maoming, Guangdong 525000, China
  • Online:2013-10-01 Published:2013-09-30

摘要: 相似性连接是数据清理工作的基本模型,获得了大量数据库工作者的关注。研究了基于编辑距离的相似性连接问题,即在两个字符串集合中寻找编辑距离小于一个阈值的字符串对,并在Pass-Join 算法的基础上,提出了一个新的Pass-Join-K 算法。Pass-Join-K 算法在长短字符串上都有很好的表现。该算法的主要思想是利用Pass-Join 算法的划分原理,以多次匹配的方式,达到更加严格地选取候选配对的目的。实验结果显示,Pass-Join-K 算法减少了候选对的数量,在实际数据集上相比元算法在运行时间上有2~5倍的提升。

关键词: 编辑距离, 相似性连接, 多次匹配, 数据清理, Pass-Join-K 算法

Abstract: Similarity join is the basic model of data cleaning in the database research and has attracted lots of attention from the database community. This paper studies the edit distance based similarity join, which finds similar strings from two large sets of strings whose edit distance is less than a given threshold, and proposes an improved Pass-Join algorithm, named Pass-Join-K. Pass-Join-K is efficient both for short strings and long strings. The main idea of Pass-Join-K is to divide the query string into more parts based on the partition strategy of Pass-Join, and filter the candidate string pairs more strictly by multi-match. The experimental results show that Pass-Join-K can decrease the candidate pairs, and run 2-5 times more quickly than the origin algorithm which outperforms state-of-the-art methods on real datasets.

Key words: edit distance, similarity join, multi-match, data cleaning, Pass-Join-K