面向两段锁并发控制的RDMA优化技术

doi:10.3778/j.issn.1673-9418.2107032

摘要/Abstract

摘要： 分布式事务的性能优化是学术界和工业界的研究热点之一。基于两段锁的并发控制技术可以保证并发事务调度的正确性，目前广泛应用于主流的商用和开源分布式数据库中。然而，现有的研究结果表明，基于传统TCP/IP协议以及Share-Nothing架构的分布式事务处理技术，受制于事务调度器的CPU低利用率、事务调度器与存取节点的网络高延迟，分布式事务的性能瓶颈明显。针对上述两个问题，提出基于远程直接数据存取（RDMA）的两段锁（2PL）并发控制优化技术，利用RDMA的高带宽、低延时以及内核旁路（消除了TCP/IP协议栈所带来的CPU开销）特性，提升分布式事务的性能。主要贡献包括基于RDMA的网络通信算子重写与优化，利用RDMA单边施加、释放读写锁时的原子性保障。基于YCSB测试基准的实验结果表明：单边排他锁算法和单边读写锁算法分别在低、高冲突负载下具有相对优越性；引入RDMA的2PL并发控制，在高冲突负载下，NO WAIT和WAIT DIE两种模式最高可分别实现5.3倍和10.6倍的吞吐量提升。

关键词: 远程直接数据存取（RDMA）, 两段锁（2PL）, 分布式数据库, 并发控制

Abstract: Performance optimization of distributed transactions is one of the hottest topics in academic and industry community. Concurrency control technology based on two-phase locking can guarantee the correctness of concurrent transaction scheduling, thus it is widely implemented in mainstream commercial and open source distributed data-bases. However, existing works show that the performance bottleneck of distributed transaction processing tech-nology, based on traditional TCP/IP protocol and Share-Nothing architecture, comes from low CPU utilization of transaction scheduler and high network latency between transaction scheduler and storage nodes. To address the above two problems, an optimization technology of two-phase locking (2PL) concurrency control based on new hardware RDMA (remote direct memory access) is proposed to improve the performance of distributed transactions by using RDMA??s characteristics, e.g., high bandwidth, low latency and kernel bypass (eliminating the CPU overhead caused by TCP/IP protocol stack). The main technical contributions of this paper include the rewriting and opti-mization of network communication operators based on RDMA, and the atomicity guarantee when using RDMA one-sidedly to acquire and release read-write locks. Experimental results based on YCSB benchmark show that: one-sided mutex lock algorithm and one-sided mutex/shared lock algorithm have relative advantages under low and high contention workload respectively; 2PL concurrency control protocols with RDMA achieve up to 5.3x and 10.6x performance gain for NO WAIT and WAIT DIE respectively under high contention workload.

Key words: remote direct memory access (RDMA), two-phase locking (2PL), distributed database, concurrency control

李婧瑶, 张倩, 赵展浩, 卢卫, 张孝, 杜小勇. 面向两段锁并发控制的RDMA优化技术[J]. 计算机科学与探索, 2023, 17(5): 1201-1209.

LI Jingyao, ZHANG Qian, ZHAO Zhanhao, LU Wei, ZHANG Xiao, DU Xiaoyong. RDMA Optimization Technology for Two-Phase Locking Concurrency Control[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(5): 1201-1209.

参考文献

[1] BERNSTEIN P A, GOODMAN N. Concurrency control in distributed database systems[J]. ACM Computing Surveys, 1981, 13(2): 185-221.
[2] STONEBRAKER M. The traditional RDBMS wisdom is (almost certainly) all wrong[EB/OL]. (2013-05-30) [2021-06-28]. https://slideshot.epfl.ch/play/suri_stonebraker.
[3] PAVLO A, CURINO C, ZDONIK S B. Skew-aware automatic database partitioning in shared-nothing, parallel OLTP sys-tems[C]//Proceedings of the 2012 ACM SIGMOD Interna-tional Conference on Management of Data, Scottsdale, May 20-24, 2012. New York: ACM, 2012: 61-72.
[4] ELMORE A J, ARORA V, TAFT R, et al. Squall: fine-grained live reconfiguration for partitioned main memory databases[C]//Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, Melbourne, May 31-Jun 4, 2015. New York: ACM, 2015: 299-313.
[5] POLYCHRONIOU O, SEN R, ROSS K A. Track join: dis-tributed joins with minimal network traffics[C]//Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data, Snowbird, Jun 22-27, 2014. New York: ACM, 2014: 1483-1494.
[6] BINNIG C, CROTTY A, GALAKATOS A, et al. The end of slow networks: it??s time for a redesign[J]. Proceedings of the VLDB Endowment, 2015, 9(7): 528-539.
[7] KALIA A, KAMINSKY M, ANDERSEN D G. Design gui-delines for high performance RDMA systems[C]//Procee-dings of the 2016 USENIX Annual Technical Conference, Denver, Jun 22-24, 2016. Berkeley: USENIX Association, 2016: 437-450.
[8] KALIA A, KAMINSKY M, ANDERSEN D G. Using RDMA efficiently for key-value services[C]//Proceedings of the 2014 ACM SIGCOMM, Chicago, Aug 17-22, 2014. New York: ACM, 2014: 295-306.
[9] ZAMANIAN E, BINNIG C, KRASKA T, et al. The end of a myth: distributed transactions can scale[J]. Proceedings of the VLDB Endowment, 2017, 10(6): 685-696.
[10] DRAGOJEVI? A, NARAYANAN D, HODSON O, et al. FaRM: fast remote memory[C]//Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implemen-tation, Seattle, Apr 2-4, 2014. Berkeley: USENIX Associa-tion, 2014: 401-414.
[11] WEI X D, DONG Z Y, CHEN R. Deconstructing RDMA-enabled distributed transactions: hybrid is better![C]//Procee-dings of the 13th USENIX Symposium on Operating Systems Design and Implementation, Carlsbad, Oct 8-10, 2018. Ber-keley: USENIX Association, 2018: 233-251.
[12] BARTHELS C, MüLLER I, TARANOV K, et al. Strong consistency is not hard to get: two-phase locking and two-phase commit on thousands of cores[J]. Proceedings of the VLDB Endowment, 2019, 12(13): 2325-2338.
[13] YOON D Y, CHOWDHURY M, MOZAFARI B. Distributed lock management with RDMA: decentralization without star-vation[C]//Proceedings of the 2018 ACM SIGMOD Interna-tional Conference on Management of Data, Houston, Jun 10-15, 2018. New York: ACM, 2018: 1571-1586.
[14] SHAMIS A, RENZELMANN M, NOVAKOVIC S, et al. Fast general distributed transactions with opacity[C]//Pro-ceedings of the 2019 ACM SIGMOD International Con-ference on Management of Data, Amsterdam, Jun 30-Jul 5, 2019. New York: ACM, 2019: 433-448.
[15] KALIA A, KAMINSKY M, ANDERSEN D G. FaSST: fast, scalable and simple distributed transactions with two-sided (RDMA) datagram RPCs[C]//Proceedings of the 12th USENIX Symposium on Operating Systems Design and Im-plementation, Savannah, Nov 2-4, 2016. Berkeley: USENIX Association, 2016: 185-201.
[16] WEI X D, SHI J X, CHEN Y Z, et al. Fast in-memory tran-saction processing using RDMA and HTM[C]//Proceedings of the 25th Symposium on Operating Systems Principles, Monterey, Oct 4-7, 2015. New York: ACM, 2015: 87-104.
[17] CHEN Y Z, WEI X D, SHI J X, et al. Fast and general distributed transactions using RDMA and HTM[C]//Procee-dings of the 11th European Conference on Computer Sys-tems, London, Apr 18-21, 2016. New York: ACM, 2016: 1-17.
[18] HARDING R, AKEN D V, PAVLO A, et al. An evaluation of distributed concurrency control[J]. Proceedings of the VLDB Endowment, 2017, 10(5): 553-564.