Abstract: Similarity join is the basic model of data cleaning in the database research and has attracted lots of attention from the database community. This paper studies the edit distance based similarity join, which finds similar strings from two large sets of strings whose edit distance is less than a given threshold, and proposes an improved Pass-Join algorithm, named Pass-Join-K. Pass-Join-K is efficient both for short strings and long strings. The main idea of Pass-Join-K is to divide the query string into more parts based on the partition strategy of Pass-Join, and filter the candidate string pairs more strictly by multi-match. The experimental results show that Pass-Join-K can decrease the candidate pairs, and run 2-5 times more quickly than the origin algorithm which outperforms state-of-the-art methods on real datasets.

Key words: edit distance, similarity join, multi-match, data cleaning, Pass-Join-K

摘要： 相似性连接是数据清理工作的基本模型，获得了大量数据库工作者的关注。研究了基于编辑距离的相似性连接问题，即在两个字符串集合中寻找编辑距离小于一个阈值的字符串对，并在Pass-Join 算法的基础上，提出了一个新的Pass-Join-K 算法。Pass-Join-K 算法在长短字符串上都有很好的表现。该算法的主要思想是利用Pass-Join 算法的划分原理，以多次匹配的方式，达到更加严格地选取候选配对的目的。实验结果显示，Pass-Join-K 算法减少了候选对的数量，在实际数据集上相比元算法在运行时间上有2~5倍的提升。

关键词: 编辑距离, 相似性连接, 多次匹配, 数据清理, Pass-Join-K 算法