关键词: 相似性连接, 权重相似性连接, 大数据, 过滤, Spark

Abstract: Similarity join is one of the key techniques in entity identification and data integration which are significant for detecting valuable information. With the development of big data, it cannot satisfy the demand of efficiency to do the job on one machine. As a consequence, distributed computation becomes a better choice to improve the    execution efficiency of similarity join. This paper gets a deeper understanding of processing algorithms for distributed similarity join based on Spark. Since the method using only suffix positional information for filtering has some shortcomings, this paper proposes a distributed similarity join processing method PSJoin, which uses the common token positional information between the prefix of one record and the suffix of another one. Also PSJoin can be      applied to the weighted case with a little change for weight tokens. It compares the weight of the first token in the mix-suffix of one record with the weights of the tokens in the other record. The weighted similarity join with PSJoin is called WTPSJoin which improves the processing efficiency. The two new methods provide novel and efficient    solutions for similarity join of big data. Experiments are tested on mixed datasets, and results show that the proposed algorithms have better performance in filtering, less running time cost and perfect speedup.

Key words: similarity join, weighted similarity join, big data, filtering, Spark