混部数据中心在线离线服务特征分析

doi:10.3778/j.issn.1673-9418.2009098

计算机科学与探索 ›› 2022, Vol. 16 ›› Issue (4): 822-834.DOI: 10.3778/j.issn.1673-9418.2009098

混部数据中心在线离线服务特征分析

陈圣蕾¹, 裘翼滔¹, 蒋从锋¹^,⁺(), 张纪林¹, 俞俊¹, 林江彬², 闫龙川³, 任祖杰⁴, 万健⁵

1.杭州电子科技大学计算机学院,杭州 310018
2.阿里云计算有限公司,杭州 311121
3.国网电力信息通信有限公司,北京 100053
4.之江实验室,杭州 311121
5.浙江科技学院信息与电子工程学院,杭州 310023

收稿日期:2020-09-07 修回日期:2020-11-02 出版日期:2022-04-01 发布日期:2020-11-06
通讯作者: + E-mail: cjiang@hdu.edu.cn
作者简介:陈圣蕾（1996—）,女,浙江缙云人,硕士研究生,主要研究方向为云计算。
裘翼滔（1995—）,男,浙江宁波人,硕士研究生,主要研究方向为云计算。
蒋从锋（1980—）,男,湖北襄阳人,博士,教授,博士生导师,主要研究方向为云计算。
张纪林（1980—）,男,山东济南人,教授,博士生导师,主要研究方向为高性能计算。
俞俊（1980—）,男,博士,教授,博士生导师,主要研究方向为机器学习。
林江彬（1983—）,男,硕士,软件工程师,主要研究方向为分布式系统。
闫龙川（1978—）,男,博士研究生,主要研究方向为绿色计算、云计算、深度学习。
任祖杰（1984—）,男,博士,主要研究方向为分布式系统。
万健（1969—）,男,博士,教授,主要研究方向为云计算、大数据分析。
基金资助:
国家自然科学基金(61972118);国家重点研发计划(2017YFB1010000);浙江省重点研发计划(2017C01SA160069)

Workload Characterization of Online and Offline Services in Co-located Data Centers

CHEN Shenglei¹, QIU Yitao¹, JIANG Congfeng¹^,⁺(), ZHANG Jilin¹, YU Jun¹, LIN Jiangbin², YAN Longchuan³, REN Zujie⁴, WAN Jian⁵

1. School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou 310018, China
2. AliCloud Computing Co., Ltd., Hangzhou 311121, China
3. State Grid Electrical Information Communication Co., Ltd., Beijing 100053, China
4. Zhejiang Lab, Hangzhou 311121, China
5. School of Information and Electronic Engineering, Zhejiang University of Science and Technology, Hangzhou 310023, China

Received:2020-09-07 Revised:2020-11-02 Online:2022-04-01 Published:2020-11-06
About author:CHEN Shenglei, born in 1996, M.S. candidate. Her research interest is cloud computing.
QIU Yitao, born in 1995, M.S. candidate. His research interest is cloud computing.
JIANG Congfeng, born in 1980, Ph.D., professor, Ph.D. supervisor. His research interest is cloud computing.
ZHANG Jilin, born in 1980, professor, Ph.D. supervisor. His research interest is high performance computing.
YU Jun, born in 1980, Ph.D., professor, Ph.D. supervisor. His research interest is machine learning.
LIN Jiangbin, born in 1983, M.S., software engineer. His research interest is distributed storage systems.
YAN Longchuan, born in 1978, Ph.D. candidate. His research interests include green computing, cloud computing and deep learning.
REN Zujie, born in 1984, Ph.D. His research interest is distributed storage systems.
WAN Jian, born in 1969, Ph.D., professor. His research interests include cloud computing and big data analytics.
Supported by:
National Natural Science Foundation of China(61972118);National Key Research and Development Program of China(2017YFB1010000);Key Research and Development Program of Zhejiang Province(2017C01SA160069)

摘要/Abstract

摘要：

为了在降低成本和减少能耗的同时提高云数据中心的资源利用率,目前许多云数据中心都采用了在线服务和离线任务混合部署的方式。虽然混合部署的方式能为数据中心带来许多益处,但它增加了任务调度的复杂性,同时对保障服务的高可靠、低延迟带来了一系列的挑战。深入分析了阿里巴巴数据中心中某一个含有4 034台服务器的集群在8天时间内所有在线服务和离线任务的运行状况。从数据分析结果中得出以下结论：首先,从在线服务的运行情况来看,所有容器的平均CPU利用率存在周期性变化,在每天的早8点到晚9点维持在一个较高水平,并且在每天凌晨4点回落到最低点。其次,对离线任务来说,除去第一天和第八天,剩下6天中任务提交峰值都集中在每天的同一时刻。95%实例的运行时间都在199 s以内,但是有0.052%的实例运行时间在1 h以上甚至会持续几天。然后,对于应用程序的相关情况,不同应用部署的容器数量存在较大差异,一个应用最多使用629个容器,最少使用1个容器。最后,对服务器、在线任务以及批处理实例进行了聚类分析,相对高资源利用率的容器占了所有容器的绝大部分,低资源利用率、短执行时间的实例则占了总实例的绝大部分。提出的发现和建议有助于数据中心管理者更详细地了解工作负载的特性,从而提高数据中心的资源利用率和各任务的容错性。

关键词: 数据中心, 工作负载特性, 在线服务, 离线任务, 调度

Abstract:

In order to reduce cost, energy consumption and improve the utilization of cloud data center resources, many cloud data centers currently use a co-allocated pattern of online services and offline batch workload. Though the co-allocated approach can bring many benefits to the data center, it adds complexity to task scheduling and brings a range of challenges such as high reliability and low latency. This paper delves into the operation of all online services and offline batch workload for the Alibaba Data Center 4034 server cluster for a period of 8 days. From the results of the data analysis, following conclusions are drawn. Firstly, from the perspective of the operation of online service, the average CPU utilization of all containers has a cyclical change, which is maintained at a high level from 8:00 am to 9:00 pm every day, and falls back to the lowest point at 4 am every day. Secondly, for offline tasks, except the first and the eighth day, the peaks of task submissions for the remaining six days are concentrated at the same time each day. The running time of 95% of the instances is within 199 s, but there are 0.052% of the instances with running time of more than one hour or even a few days. Thirdly, for the application-related situation, there are large differences in the number of containers deployed in different applications. One application uses up to 629 containers and at least 1 container. Finally, cluster analysis is conducted on servers, online tasks and batch instances. Containers with relatively high resource utilization account for the vast majority of all containers, while instances with low resource utilization and short execution time account for the vast majority of all instances. The findings and recommendations in this paper can help data center managers understand the characteristics of co-located workloads more detailedly, thereby improving resource utilization and fault tolerance for each task.

Key words: data center, workload characteristics, online service, offline task, scheduling

中图分类号:

TP393

陈圣蕾, 裘翼滔, 蒋从锋, 张纪林, 俞俊, 林江彬, 闫龙川, 任祖杰, 万健. 混部数据中心在线离线服务特征分析[J]. 计算机科学与探索, 2022, 16(4): 822-834.

CHEN Shenglei, QIU Yitao, JIANG Congfeng, ZHANG Jilin, YU Jun, LIN Jiangbin, YAN Longchuan, REN Zujie, WAN Jian. Workload Characterization of Online and Offline Services in Co-located Data Centers[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(4): 822-834.

图/表 29

图1 混部集群架构

Fig.1 Mixed cluster architecture

表1 阿里巴巴数据集记录行数

Table 1 Alibaba dataset record line number

表名	记录数	文件大小
machine meta	17 592	0.56 MB
machine usage	246 637 252	8.38 GB
container meta	370 540	22.26 MB
container usage	4 015 763 787	164.09 GB
batch task	14 295 731	0.98 GB
batch instance	1 351 255 775	103.69 GB

图2 离线任务上三级任务模式关系图

Fig.2 Three-level task mode relationship diagram on offline tasks

图3 每台服务器包含的容器数量

Fig.3 Container amount of server

图4 不同的容器数量对应的服务器数量

Fig.4 The number of servers corresponding todifferent container number

表2 只出现一种状态的容器数量

Table 2 Container amount with only one state

状态	started	stopped	allocated	unknow	总计
容器数量	70 903	400	39	0	71 342

表3 出现过两种状态的容器数量

Table 3 Container amount with two states

状态	started,stopped	started,allocated	started,unknow	总计
容器数量	118	10	5	133

表4 出现过三种状态的容器数量

Table 4 Container amount with three states

状态	started,allocated,stopped
容器数量	1

图5 不同内存请求量对应的容器数量

Fig.5 Container amount corresponding to different memory requests

图6 在线任务的资源利用率随时间变化情况（第2天至第9天）

Fig.6 Resource usage of online tasks changes over time (Day 2—9)

图7 不同平均资源利用率的容器数量分布

Fig.7 Distribution of the number of containers with different average resource usage

表5 容器资源利用率分布拟合函数以及参数

Table 5 Fitting function and parameter value of container resource usage distribution

Fitting function	Resource category	$a$	$b$	$c$	R-square
$y = a exp (b x)$	CPU	5 792	-0.086	—	0.977
$y = a exp (b x)$	memory	0.003	0.146	—	0.617
	disk_io	14 430	8.203	2.452	0.992

表5 容器资源利用率分布拟合函数以及参数

Table 5 Fitting function and parameter value of container resource usage distribution

Fitting function	Resource category	$a$	$b$	$c$	R-square
$y = a exp (b x)$	CPU	5 792	-0.086	—	0.977
$y = a exp (b x)$	memory	0.003	0.146	—	0.617
	disk_io	14 430	8.203	2.452	0.992

图8 实例的平均CPU核数使用个数分布图

Fig.8 Distribution of average number of CPU cores used by instance

图9 实例的平均内存利用率

Fig.9 Average memory usage of instance

图10 所有实例的运行时间

Fig.10 Duration of instances

图11 每小时内到达的批处理作业数量

Fig.11 Number of jobs arriving in an hour

图12 服务器在不同分类数下的评价分数

Fig.12 Evaluation score of servers under different classification number

图13 服务器三维聚类效果图

Fig.13 3D clustering rendering of servers

表6 所有服务器特征指标的边界

Table 6 Boundaries of feature vectors for servers

Feature vectorgroup name	Average CPU	Average memory	Average disk
mGroup0	35.775~44.424	81.836~92.608	3.298~32.604
mGroup1	0.000 2~28.393	2.999~48.931	0~25.944
mGroup2	2.257~21.418	52.162~96.156	0.611~24.015
mGroup3	39.908~60.559	81.562~92.383	2.940~28.120
mGroup4	34.719~58.793	81.577~92.079	40.815~98.108
mGroup5	20.179~37.649	49.439~94.623	2.737~20.349

图14 每个分类中服务器的数量

Fig.14 Servers amount in each group

图15 服务器每个分类的平均资源利用率

Fig.15 Average resource usage for server groups

图16 容器在不同分类数下的评价分数

Fig.16 Evaluation scores of containers under different classification number

图17 容器三维聚类效果图

Fig.17 3D clustering rendering of containers

表7 所有容器特征指标的边界

Table 7 Boundaries of feature vectors for containers

Feature vectorgroup name	Average CPU	Average memory	Average disk
cGroup0	0.000 1~99.989	64.208 5~100.000	0.628~98.897
cGroup1	0~99.999 8	1.008~69.847	0~98.910 2

图18 在线服务的平均资源利用率CDF图

Fig.18 CDF of average resource usage for online service groups

图19 抽样实例在不同聚类数下的评价

Fig.19 Evaluation of sampling instances under different number of clusters

表8 抽样实例特征指标的边界

Table 8 Boundaries of feature vectors for instance

Feature vector group name	Average CPU	Average memory	Duration/s
iGroup0	0~4 257	0~11.130 0	271~221 229
iGroup1	0~2 135	0~91.599 9	1~296

图20 两类实例的平均资源利用率CDF图

Fig.20 CDF of average resource usage for batch instance groups

表9 每类服务器中两类容器及两类实例的数量占比

Table 9 Proportion of two types of containers and instances in each type of server

Feature vector group name	cGroup0	cGroup1	iGroup0	iGroup1
mGroup0	0.74	0.26	0.03	0.97
mGroup1	0.75	0.25	0.03	0.97
mGroup2	0.79	0.21	0.03	0.97
mGroup3	0.81	0.19	0.04	0.96
mGroup4	0.78	0.22	0.04	0.96
mGroup5	0.67	0.33	0.03	0.97

参考文献 33

[1]	JYOTHI S A, CURINO C, MENACHE I, et al. Morpheus: towards automated SLOs for enterprise clusters[C]// Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation, Savannah, Nov 2-4, 2016. Berkeley: USENIX Association, 2016: 117-134.
[2]	RAJAN K, KAKADIA D, CURINO C, et al. PerfOrator: eloquent performance models for resource optimization[C]// Proceedings of the 7th ACM Symposium on Cloud Computing, Santa Clara, Oct 5-7, 2016. New York: ACM, 2016: 415-427.
[3]	XU G Y, XU C Z. Prometheus: online estimation of optimal memory demands for workers in in-memory distributed computation[C]// Proceedings of the 2017 Symposium on Cloud Computing, Santa Clara, Sep 24-27, 2017. New York: ACM, 2017: 655-655.
[4]	REISS C, TUMANOV A, GANGER G R, et al. Towards understanding heterogeneous clouds at scale: Google trace analysis: ISTC-CC-TR-12-101[R]. Pittsburgh: Carnegie Mellon University, 2012.
[5]	ZHOU M S, DONG X S, CHEN H, et al. Fine-grained scheduling in multi-resource clusters[J]. The Journal of Supercomputing, 2020, 76(3):1931-1958. DOI URL
[6]	ZOU D Q, QIAN S Y, XUE G T, et al. UpPreempt: a fine-grained preemptive scheduling strategy for container-based clusters[C]// Proceedings of the 24th IEEE International Conference on Parallel and Distributed Systems, Singapore, Dec 11-13, 2018. Piscataway: IEEE, 2018: 373-380.
[7]	BI J, YUAN H T, TAN W, et al. Application-aware dynamic fine-grained resource provisioning in a virtualized cloud data center[J]. IEEE Transactions on Automation Science and Engineering, 2017, 14(2):1172-1184. DOI URL
[8]	USMANI Z, SINGH S. A survey of virtual machine placement techniques in a cloud data center[J]. Procedia Computer Science, 2016, 78:491-498. DOI URL
[9]	AHMAD R W, GANI A, HAMId S H A, et al. A survey on virtual machine migration and server consolidation frameworks for cloud data centers[J]. Journal of Network and Computer Applications, 2015, 52:11-25. DOI URL
[10]	TOSATTO A, RUIU P, ATTANASIO A. Container-based orchestration in cloud: state of the art and challenges[C]// Proceedings of the 9th International Conference on Complex, Intelligent, and Software Intensive Systems, Santa Catarina, Jul 8-10, 2015. Washington: IEEE Computer Society, 2015: 70-75.
[11]	GOUDARZI H, PEDRAM M. Hierarchical SLA-driven resource management for peak power-aware and energy-efficient operation of a cloud datacenter[J]. IEEE Transactions on Cloud Computing, 2016, 4(2):222-236. DOI URL
[12]	PETRUCCI V, LAURENZANO M A, DOHERTY J, et al. Octopus-Man: QoS-driven task management for heterogeneous multicores in warehouse-scale computers[C]// Proceedings of the 21st IEEE International Symposium on High Performance Computer Architecture, Burlingame, Feb 7-11, 2015. Washi- ngton: IEEE Computer Society, 2015: 246-258.
[13]	CHEN W, RAO J, ZHOU X B. Preemptive, low latency datacenter scheduling via lightweight virtualization[C]// Proceedings of the 2017 USENIX Annual Technical Conference, Santa Clara, Jul 12-14, 2017. Berkeley: USENIX Association, 2017: 251-263.
[14]	YAN Y, GAO Y J, CHEN Y, et al. TR-spark: transient computing for big data analytics[C]// Proceedings of the 7th ACM Symposium on Cloud Computing, Santa Clara, Oct 5-7, 2016. New York: ACM, 2016: 484-496.
[15]	CHEN S, DELIMITROU C, MARTÍNEZ J F. Parties: Qos-aware resource partitioning for multiple interactive services[C]// Proceedings of the 24th International Conference on Architectural Support for Programming Languages and Operating Systems, Providence, Apr 13-17, 2019. New York: ACM, 2019: 107-120.
[16]	ISLAM M A, GANDHI A, REN S L. Minimizing electricity cost for geo-distributed interactive services with tail latency constraint[C]// Proceedings of the 7th International Green and Sustainable Computing Conference, Hangzhou, Nov 7-9, 2016. Washington: IEEE Computer Society, 2016: 1-8.
[17]	CHENG Y, ANWAR A, DUAN X J. Analyzing Alibaba’s co-located datacenter workloads[C]// Proceedings of the 2018 IEEE International Conference on Big Data, Seattle, Dec 10-13, 2018. Piscataway: IEEE, 2018: 292-297.
[18]	ZHANG Z, LI C, TAO Y Y, et al. Fuxi: a fault-tolerant resource management and job scheduling system at Internet scale[J]. Proceedings of the VLDB Endowment, 2014, 7(13):1393-1404. DOI URL
[19]	Pouch container engine[EB/OL]. [2020-06-23]. https://github.com/Alibaba/pouch.
[20]	Alibaba trace[EB/OL]. [2020-06-23]. https://github.com/alibaba/clusterdata.
[21]	DENG L, REN Y L, XU F, et al. Resource utilization analysis of Alibaba cloud[C]// LNCS 10954: Proceedings of the 14th International Conference on Intelligent Computing Theories and Application, Wuhan, Aug 15-18, 2018. Cham: Springer, 2018: 183-194.
[22]	ARORA P, VARSHNEY S. Analysis of k-means and k-medoids algorithm for big data[J]. Procedia Computer Science, 2016, 78:507-512. DOI URL
[23]	ŁUKASIK S, KOWALSKI P A, CHARYTANOWICZ M, et al. Clustering using flower pollination algorithm and Calinski-Harabasz index[C]// Proceedings of the 2016 IEEE Congress on Evolutionary Computation, Vancouver, Jul 24-29, 2016. Piscataway: IEEE, 2016: 2724-2728.
[24]	SHISHIRA S R, KANDASAMY A, CHANDRASEKARAN K. Workload characterization: survey of current approaches and research challenges[C]// Proceedings of the 7th International Conference on Computer and Communication Technology, Allahabad, Nov 24-26, 2017. New York: ACM, 2017: 151-156.
[25]	Google trace[EB/OL]. [2020-06-23]. https://github.com/google/cluster-data.
[26]	REISS C, TUMANOV A, GANGER G R, et al. Heterogeneity and dynamicity of clouds at scale: Google trace analysis[C]// Proceedings of the 3rd ACM Symposium on Cloud Computing. New York: ACM, 2012: 7.
[27]	FAN Z W, HUANG P J, HUANG P S, et al. A feature generation framework for Google trace analysis[C]// Proceedings of the 2015 International Conference on Machine Learning and Cybernetics. Piscataway: IEEE, 2015: 229-234.
[28]	LU C Z, YE K J, XU G Y, et al. Imbalance in the cloud: an analysis on Alibaba cluster trace[C]// Proceedings of the 2017 IEEE International Conference on Big Data, Boston, Dec 11-14, 2017. Washington: IEEE Computer Society, 2017: 2884-2892.
[29]	CHENG Y, CHAI Z, ANWAR A, Characterizing co-located datacenter workloads: an Alibaba case study[J]. arXiv: 1808. 02919, 2018.
[30]	LIU Q X, YU Z B. The elasticity and plasticity in semi-containerized co-locating cloud workload: a view from Alibaba trace[C]// Proceedings of the 2018 ACM Symposium on Cloud Computing, Carlsbad, Oct 11-13, 2018. New York: ACM, 2018: 347-360.
[31]	CHEN Y, GANAPATHI A S, GRIFFITH R, et al. Analysis and lessons from a publicly available Google cluster trace: UCB/EECS-2010-95[R]. Berkeley: University of California, 2010.
[32]	ALAM M, SHAKIL K A, SETHI S. Analysis and clustering of workload in Google cluster trace based on resource usage[C]// Proceedings of the 2016 IEEE International Conference on Computational Science and Engineering, and IEEE International Conference on Embedded and Ubiquitous Computing, and 15th International Symposium on Distributed Computing and Applications for Business Engineering, Paris, Aug 24-26, 2016. Washington: IEEE Computer Society, 2016: 740-747.
[33]	CHEN W Y, YE K J, WANG Y, et al. How does the workload look like in production cloud? Analysis and clustering of workloads on Alibaba cluster trace[C]// Proceedings of the 24th IEEE International Conference on Parallel and Distributed Systems, Singapore, Dec 11-13, 2018. Piscataway: IEEE, 2018: 102-109.

混部数据中心在线离线服务特征分析

Workload Characterization of Online and Offline Services in Co-located Data Centers

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 29

参考文献 33

相关文章 15

编辑推荐

Metrics

[1]	余达明, 张震. FSDC：灵活的高可扩展数据中心网络结构[J]. 计算机科学与探索, 2022, 16(4): 855-864.
[2]	官铮, 胡扬, 杨志军, 何敏. 分布式WLAN全双工链路加权调度算法[J]. 计算机科学与探索, 2022, 16(2): 372-383.
[3]	李成严, 宋月, 马金涛. 模糊云资源调度问题的RIOPSO算法[J]. 计算机科学与探索, 2021, 15(8): 1534-1545.
[4]	叶进, 谢紫琪, 肖庆宇, 宋玲, 李晓欢. 数据中心网络中基于ELM的流簇大小推理机制[J]. 计算机科学与探索, 2021, 15(2): 261-269.
[5]	郭羽含，伊鹏. 车辆合乘问题的分布式复合变邻域搜索算法[J]. 计算机科学与探索, 2019, 13(2): 330-341.
[6]	孙怀英，虞慧群，范贵生，陈丽琼. 支持SDN的Hadoop中的时间最小化任务调度[J]. 计算机科学与探索, 2018, 12(11): 1767-1776.
[7]	裴树军，宋冬梅，孔德凯. Map/Reduce下快速剪枝算法在复杂任务调度中的应用[J]. 计算机科学与探索, 2018, 12(1): 72-81.
[8]	余雅君，刘峥，徐明伟. 数据中心网络TCP Incast问题研究[J]. 计算机科学与探索, 2017, 11(9): 1361-1378.
[9]	李芳芳，刘冲，于戈. 面向CPS的时间戳不确定事件调度算法[J]. 计算机科学与探索, 2017, 11(6): 887-896.
[10]	张忆文，王成. 可靠性感知周期任务能耗管理调度算法[J]. 计算机科学与探索, 2017, 11(5): 833-841.
[11]	张飞朋，陈琳，张京京. 面向大规模复杂网络测量和性能瓶颈分析方法[J]. 计算机科学与探索, 2017, 11(2): 262-270.
[12]	王德胜，张伟哲，郝萌，鲁刚钊，白恩慈. 云计算环境中虚拟机内存自适应调节算法研究[J]. 计算机科学与探索, 2017, 11(1): 70-79.
[13]	朱鹏，何琨，曹伟刚，杨欢. 基于穴度的三维时空优化问题的贪心调度算法[J]. 计算机科学与探索, 2016, 10(8): 1051-1062.
[14]	高任飞，武继刚，周莹，张耀国. 极小通讯延迟的虚拟机分配算法[J]. 计算机科学与探索, 2016, 10(7): 924-935.
[15]	周丹，葛洪伟，苏树智，袁运浩. 基于紧凑度和调度处理的粒子群优化算法[J]. 计算机科学与探索, 2016, 10(5): 742-750.