关键词: 主题标引, YARN平台, 有向无环图计算框架, 固态硬盘

Abstract: Subject indexing is a very important component in personalized intelligent search system. However, the huge amount of data resource makes it a great challenge in processing performance. Nowadays, the subject indexing over MapReduce computing framework has been widely used, which has shortcomings, such as time-consuming of starting the tasks and too many disk IOs. This paper adopts YARN (yet another resource negotiator) as the underlying platform, and chooses more appropriate calculation frameworks to improve the performance. For the feature of subject indexing algorithm, which is multistage, the directed acyclic graph (DAG) model is selected to avoid unnecessary operations of job split, which reduces the disk IOs of intermediate results. In addition, considering the sorting strategy is time-consuming, this paper adopts Hash-based data gathering strategy to improve computing performance. However, the new policy will bring the problem of random read. This paper designs an optimization strategy, which takes advantage of the feature of high-speed random read of solid state disk (SSD), to further improve the computational efficiency. Through the experimental results, choosing targeted computing framework based on YARN and optimizing it, can effectively improve computing performance.

Key words: subject indexing, YARN platform, directed acyclic graph (DAG) computation, solid state disk