关键词: RDD数据块, 缓存增益, 缓存替换, Spark处理框架

Abstract: Cache replacement is a hot and difficult research topic in the field of memory optimization of Spark. However, the diversity of application characteristics, limited memory resources, and the uncertainty of cache replacement present challenges to achieving high system execution performance. Inefficient cache replacement strategies may lead to various performance issues, such as long application execution time and low resource utilization. Based on this, researchers proposes a gain-aware cache replacement strategy for RDD data blocks on Spark. This strategy first establishes a cache value assessment model that comprehensively considers partition size, reference count, computation cost, and resource cost impact factors of data blocks to accurately assess the cache value of data blocks. Then, a cache gain problem model is proposed to formalize the optimization problem of cache management. Finally, a gain-aware cache replacement algorithm for RDD data blocks (CRCA) is proposed to ensure the maximization of cache benefits of RDD data blocks in memory. To validate the effectiveness of the CRCA algorithm, we built a real big data cluster experimental platform based on Spark and conducted experimental evaluations using diversified workloads from the HiBench benchmark suite The results show that the proposed cache replacement algorithm (CRCA) outperforms the existing least recently used(LRU) and least partition weight(LPW) algorithms in terms of task execution time and CPU utilization.

Key words: RDD data blocks, cache gain, cache replacement, Spark framework