关键词: MapReduce, 行列混合存储, 延迟物化, 多表连接优化

Abstract: The MapReduce technology has become one of the key technology for massive data processing. However, the limitation of its computing framework leads to the poor performance in multi-join query analysis tasks. To deal with this problem, this paper proposes an adaptive multi-join optimization method for MapReduce framework, called MHMO (MapReduce based heuristic multi-join optimization). For a given query including multi-join, this paper first constructs the join graph to judge its join pattern, then recommends the “optimal” execution strategy for different patterns. Particularly, for hybrid join, this paper first converts and divides it into a set of simple join patterns, then defines the cost model to choose the execution order between different groups with minimum cost. Integrated with the row-column storage and deferred materialized technology, MHMO can improve the multi-join performance in MapReduce framework significantly. Finally, based on the benchmark dataset TPCH, several experiments are made to testify the effectiveness of MHMO.

Key words: MapReduce, row-column storage, deferred materialized, multi-join optimization