Workload Characterization of Online and Offline Services in Co-located Data Centers

doi:10.3778/j.issn.1673-9418.2009098

Journal of Frontiers of Computer Science and Technology ›› 2022, Vol. 16 ›› Issue (4): 822-834.DOI: 10.3778/j.issn.1673-9418.2009098

• Database Technology • Previous Articles Next Articles

Workload Characterization of Online and Offline Services in Co-located Data Centers

CHEN Shenglei¹, QIU Yitao¹, JIANG Congfeng¹^,⁺(), ZHANG Jilin¹, YU Jun¹, LIN Jiangbin², YAN Longchuan³, REN Zujie⁴, WAN Jian⁵

1. School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou 310018, China
2. AliCloud Computing Co., Ltd., Hangzhou 311121, China
3. State Grid Electrical Information Communication Co., Ltd., Beijing 100053, China
4. Zhejiang Lab, Hangzhou 311121, China
5. School of Information and Electronic Engineering, Zhejiang University of Science and Technology, Hangzhou 310023, China

Received:2020-09-07 Revised:2020-11-02 Online:2022-04-01 Published:2020-11-06
About author:CHEN Shenglei, born in 1996, M.S. candidate. Her research interest is cloud computing.
QIU Yitao, born in 1995, M.S. candidate. His research interest is cloud computing.
JIANG Congfeng, born in 1980, Ph.D., professor, Ph.D. supervisor. His research interest is cloud computing.
ZHANG Jilin, born in 1980, professor, Ph.D. supervisor. His research interest is high performance computing.
YU Jun, born in 1980, Ph.D., professor, Ph.D. supervisor. His research interest is machine learning.
LIN Jiangbin, born in 1983, M.S., software engineer. His research interest is distributed storage systems.
YAN Longchuan, born in 1978, Ph.D. candidate. His research interests include green computing, cloud computing and deep learning.
REN Zujie, born in 1984, Ph.D. His research interest is distributed storage systems.
WAN Jian, born in 1969, Ph.D., professor. His research interests include cloud computing and big data analytics.
Supported by:
National Natural Science Foundation of China(61972118);National Key Research and Development Program of China(2017YFB1010000);Key Research and Development Program of Zhejiang Province(2017C01SA160069)

混部数据中心在线离线服务特征分析

陈圣蕾¹, 裘翼滔¹, 蒋从锋¹^,⁺(), 张纪林¹, 俞俊¹, 林江彬², 闫龙川³, 任祖杰⁴, 万健⁵

1.杭州电子科技大学计算机学院,杭州 310018
2.阿里云计算有限公司,杭州 311121
3.国网电力信息通信有限公司,北京 100053
4.之江实验室,杭州 311121
5.浙江科技学院信息与电子工程学院,杭州 310023

通讯作者: + E-mail: cjiang@hdu.edu.cn
作者简介:陈圣蕾（1996—）,女,浙江缙云人,硕士研究生,主要研究方向为云计算。
裘翼滔（1995—）,男,浙江宁波人,硕士研究生,主要研究方向为云计算。
蒋从锋（1980—）,男,湖北襄阳人,博士,教授,博士生导师,主要研究方向为云计算。
张纪林（1980—）,男,山东济南人,教授,博士生导师,主要研究方向为高性能计算。
俞俊（1980—）,男,博士,教授,博士生导师,主要研究方向为机器学习。
林江彬（1983—）,男,硕士,软件工程师,主要研究方向为分布式系统。
闫龙川（1978—）,男,博士研究生,主要研究方向为绿色计算、云计算、深度学习。
任祖杰（1984—）,男,博士,主要研究方向为分布式系统。
万健（1969—）,男,博士,教授,主要研究方向为云计算、大数据分析。
基金资助:
国家自然科学基金(61972118);国家重点研发计划(2017YFB1010000);浙江省重点研发计划(2017C01SA160069)

Abstract

Abstract:

In order to reduce cost, energy consumption and improve the utilization of cloud data center resources, many cloud data centers currently use a co-allocated pattern of online services and offline batch workload. Though the co-allocated approach can bring many benefits to the data center, it adds complexity to task scheduling and brings a range of challenges such as high reliability and low latency. This paper delves into the operation of all online services and offline batch workload for the Alibaba Data Center 4034 server cluster for a period of 8 days. From the results of the data analysis, following conclusions are drawn. Firstly, from the perspective of the operation of online service, the average CPU utilization of all containers has a cyclical change, which is maintained at a high level from 8:00 am to 9:00 pm every day, and falls back to the lowest point at 4 am every day. Secondly, for offline tasks, except the first and the eighth day, the peaks of task submissions for the remaining six days are concentrated at the same time each day. The running time of 95% of the instances is within 199 s, but there are 0.052% of the instances with running time of more than one hour or even a few days. Thirdly, for the application-related situation, there are large differences in the number of containers deployed in different applications. One application uses up to 629 containers and at least 1 container. Finally, cluster analysis is conducted on servers, online tasks and batch instances. Containers with relatively high resource utilization account for the vast majority of all containers, while instances with low resource utilization and short execution time account for the vast majority of all instances. The findings and recommendations in this paper can help data center managers understand the characteristics of co-located workloads more detailedly, thereby improving resource utilization and fault tolerance for each task.

Key words: data center, workload characteristics, online service, offline task, scheduling

摘要：

为了在降低成本和减少能耗的同时提高云数据中心的资源利用率,目前许多云数据中心都采用了在线服务和离线任务混合部署的方式。虽然混合部署的方式能为数据中心带来许多益处,但它增加了任务调度的复杂性,同时对保障服务的高可靠、低延迟带来了一系列的挑战。深入分析了阿里巴巴数据中心中某一个含有4 034台服务器的集群在8天时间内所有在线服务和离线任务的运行状况。从数据分析结果中得出以下结论：首先,从在线服务的运行情况来看,所有容器的平均CPU利用率存在周期性变化,在每天的早8点到晚9点维持在一个较高水平,并且在每天凌晨4点回落到最低点。其次,对离线任务来说,除去第一天和第八天,剩下6天中任务提交峰值都集中在每天的同一时刻。95%实例的运行时间都在199 s以内,但是有0.052%的实例运行时间在1 h以上甚至会持续几天。然后,对于应用程序的相关情况,不同应用部署的容器数量存在较大差异,一个应用最多使用629个容器,最少使用1个容器。最后,对服务器、在线任务以及批处理实例进行了聚类分析,相对高资源利用率的容器占了所有容器的绝大部分,低资源利用率、短执行时间的实例则占了总实例的绝大部分。提出的发现和建议有助于数据中心管理者更详细地了解工作负载的特性,从而提高数据中心的资源利用率和各任务的容错性。

关键词: 数据中心, 工作负载特性, 在线服务, 离线任务, 调度

CLC Number:

TP393

CHEN Shenglei, QIU Yitao, JIANG Congfeng, ZHANG Jilin, YU Jun, LIN Jiangbin, YAN Longchuan, REN Zujie, WAN Jian. Workload Characterization of Online and Offline Services in Co-located Data Centers[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(4): 822-834.

陈圣蕾, 裘翼滔, 蒋从锋, 张纪林, 俞俊, 林江彬, 闫龙川, 任祖杰, 万健. 混部数据中心在线离线服务特征分析[J]. 计算机科学与探索, 2022, 16(4): 822-834.

Figures/Tables 29

Fig.1 Mixed cluster architecture

Table 1 Alibaba dataset record line number

表名	记录数	文件大小
machine meta	17 592	0.56 MB
machine usage	246 637 252	8.38 GB
container meta	370 540	22.26 MB
container usage	4 015 763 787	164.09 GB
batch task	14 295 731	0.98 GB
batch instance	1 351 255 775	103.69 GB

Fig.2 Three-level task mode relationship diagram on offline tasks

Fig.3 Container amount of server

Fig.4 The number of servers corresponding todifferent container number

Table 2 Container amount with only one state

状态	started	stopped	allocated	unknow	总计
容器数量	70 903	400	39	0	71 342

Table 3 Container amount with two states

状态	started,stopped	started,allocated	started,unknow	总计
容器数量	118	10	5	133

Table 4 Container amount with three states

状态	started,allocated,stopped
容器数量	1

Fig.5 Container amount corresponding to different memory requests

Fig.6 Resource usage of online tasks changes over time (Day 2—9)

Fig.7 Distribution of the number of containers with different average resource usage

Table 5 Fitting function and parameter value of container resource usage distribution

Fitting function	Resource category	$a$	$b$	$c$	R-square
$y = a exp (b x)$	CPU	5 792	-0.086	—	0.977
$y = a exp (b x)$	memory	0.003	0.146	—	0.617
	disk_io	14 430	8.203	2.452	0.992

Table 5 Fitting function and parameter value of container resource usage distribution

Fitting function	Resource category	$a$	$b$	$c$	R-square
$y = a exp (b x)$	CPU	5 792	-0.086	—	0.977
$y = a exp (b x)$	memory	0.003	0.146	—	0.617
	disk_io	14 430	8.203	2.452	0.992

Fig.8 Distribution of average number of CPU cores used by instance

Fig.9 Average memory usage of instance

Fig.10 Duration of instances

Fig.11 Number of jobs arriving in an hour

Fig.12 Evaluation score of servers under different classification number

Fig.13 3D clustering rendering of servers

Table 6 Boundaries of feature vectors for servers

Feature vectorgroup name	Average CPU	Average memory	Average disk
mGroup0	35.775~44.424	81.836~92.608	3.298~32.604
mGroup1	0.000 2~28.393	2.999~48.931	0~25.944
mGroup2	2.257~21.418	52.162~96.156	0.611~24.015
mGroup3	39.908~60.559	81.562~92.383	2.940~28.120
mGroup4	34.719~58.793	81.577~92.079	40.815~98.108
mGroup5	20.179~37.649	49.439~94.623	2.737~20.349

Fig.14 Servers amount in each group

Fig.15 Average resource usage for server groups

Fig.16 Evaluation scores of containers under different classification number

Fig.17 3D clustering rendering of containers

Table 7 Boundaries of feature vectors for containers

Feature vectorgroup name	Average CPU	Average memory	Average disk
cGroup0	0.000 1~99.989	64.208 5~100.000	0.628~98.897
cGroup1	0~99.999 8	1.008~69.847	0~98.910 2

Fig.18 CDF of average resource usage for online service groups

Fig.19 Evaluation of sampling instances under different number of clusters

Table 8 Boundaries of feature vectors for instance

Feature vector group name	Average CPU	Average memory	Duration/s
iGroup0	0~4 257	0~11.130 0	271~221 229
iGroup1	0~2 135	0~91.599 9	1~296

Fig.20 CDF of average resource usage for batch instance groups

Table 9 Proportion of two types of containers and instances in each type of server

Feature vector group name	cGroup0	cGroup1	iGroup0	iGroup1
mGroup0	0.74	0.26	0.03	0.97
mGroup1	0.75	0.25	0.03	0.97
mGroup2	0.79	0.21	0.03	0.97
mGroup3	0.81	0.19	0.04	0.96
mGroup4	0.78	0.22	0.04	0.96
mGroup5	0.67	0.33	0.03	0.97

References 33

[1]	JYOTHI S A, CURINO C, MENACHE I, et al. Morpheus: towards automated SLOs for enterprise clusters[C]// Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation, Savannah, Nov 2-4, 2016. Berkeley: USENIX Association, 2016: 117-134.
[2]	RAJAN K, KAKADIA D, CURINO C, et al. PerfOrator: eloquent performance models for resource optimization[C]// Proceedings of the 7th ACM Symposium on Cloud Computing, Santa Clara, Oct 5-7, 2016. New York: ACM, 2016: 415-427.
[3]	XU G Y, XU C Z. Prometheus: online estimation of optimal memory demands for workers in in-memory distributed computation[C]// Proceedings of the 2017 Symposium on Cloud Computing, Santa Clara, Sep 24-27, 2017. New York: ACM, 2017: 655-655.
[4]	REISS C, TUMANOV A, GANGER G R, et al. Towards understanding heterogeneous clouds at scale: Google trace analysis: ISTC-CC-TR-12-101[R]. Pittsburgh: Carnegie Mellon University, 2012.
[5]	ZHOU M S, DONG X S, CHEN H, et al. Fine-grained scheduling in multi-resource clusters[J]. The Journal of Supercomputing, 2020, 76(3):1931-1958. DOI URL
[6]	ZOU D Q, QIAN S Y, XUE G T, et al. UpPreempt: a fine-grained preemptive scheduling strategy for container-based clusters[C]// Proceedings of the 24th IEEE International Conference on Parallel and Distributed Systems, Singapore, Dec 11-13, 2018. Piscataway: IEEE, 2018: 373-380.
[7]	BI J, YUAN H T, TAN W, et al. Application-aware dynamic fine-grained resource provisioning in a virtualized cloud data center[J]. IEEE Transactions on Automation Science and Engineering, 2017, 14(2):1172-1184. DOI URL
[8]	USMANI Z, SINGH S. A survey of virtual machine placement techniques in a cloud data center[J]. Procedia Computer Science, 2016, 78:491-498. DOI URL
[9]	AHMAD R W, GANI A, HAMId S H A, et al. A survey on virtual machine migration and server consolidation frameworks for cloud data centers[J]. Journal of Network and Computer Applications, 2015, 52:11-25. DOI URL
[10]	TOSATTO A, RUIU P, ATTANASIO A. Container-based orchestration in cloud: state of the art and challenges[C]// Proceedings of the 9th International Conference on Complex, Intelligent, and Software Intensive Systems, Santa Catarina, Jul 8-10, 2015. Washington: IEEE Computer Society, 2015: 70-75.
[11]	GOUDARZI H, PEDRAM M. Hierarchical SLA-driven resource management for peak power-aware and energy-efficient operation of a cloud datacenter[J]. IEEE Transactions on Cloud Computing, 2016, 4(2):222-236. DOI URL
[12]	PETRUCCI V, LAURENZANO M A, DOHERTY J, et al. Octopus-Man: QoS-driven task management for heterogeneous multicores in warehouse-scale computers[C]// Proceedings of the 21st IEEE International Symposium on High Performance Computer Architecture, Burlingame, Feb 7-11, 2015. Washi- ngton: IEEE Computer Society, 2015: 246-258.
[13]	CHEN W, RAO J, ZHOU X B. Preemptive, low latency datacenter scheduling via lightweight virtualization[C]// Proceedings of the 2017 USENIX Annual Technical Conference, Santa Clara, Jul 12-14, 2017. Berkeley: USENIX Association, 2017: 251-263.
[14]	YAN Y, GAO Y J, CHEN Y, et al. TR-spark: transient computing for big data analytics[C]// Proceedings of the 7th ACM Symposium on Cloud Computing, Santa Clara, Oct 5-7, 2016. New York: ACM, 2016: 484-496.
[15]	CHEN S, DELIMITROU C, MARTÍNEZ J F. Parties: Qos-aware resource partitioning for multiple interactive services[C]// Proceedings of the 24th International Conference on Architectural Support for Programming Languages and Operating Systems, Providence, Apr 13-17, 2019. New York: ACM, 2019: 107-120.
[16]	ISLAM M A, GANDHI A, REN S L. Minimizing electricity cost for geo-distributed interactive services with tail latency constraint[C]// Proceedings of the 7th International Green and Sustainable Computing Conference, Hangzhou, Nov 7-9, 2016. Washington: IEEE Computer Society, 2016: 1-8.
[17]	CHENG Y, ANWAR A, DUAN X J. Analyzing Alibaba’s co-located datacenter workloads[C]// Proceedings of the 2018 IEEE International Conference on Big Data, Seattle, Dec 10-13, 2018. Piscataway: IEEE, 2018: 292-297.
[18]	ZHANG Z, LI C, TAO Y Y, et al. Fuxi: a fault-tolerant resource management and job scheduling system at Internet scale[J]. Proceedings of the VLDB Endowment, 2014, 7(13):1393-1404. DOI URL
[19]	Pouch container engine[EB/OL]. [2020-06-23]. https://github.com/Alibaba/pouch.
[20]	Alibaba trace[EB/OL]. [2020-06-23]. https://github.com/alibaba/clusterdata.
[21]	DENG L, REN Y L, XU F, et al. Resource utilization analysis of Alibaba cloud[C]// LNCS 10954: Proceedings of the 14th International Conference on Intelligent Computing Theories and Application, Wuhan, Aug 15-18, 2018. Cham: Springer, 2018: 183-194.
[22]	ARORA P, VARSHNEY S. Analysis of k-means and k-medoids algorithm for big data[J]. Procedia Computer Science, 2016, 78:507-512. DOI URL
[23]	ŁUKASIK S, KOWALSKI P A, CHARYTANOWICZ M, et al. Clustering using flower pollination algorithm and Calinski-Harabasz index[C]// Proceedings of the 2016 IEEE Congress on Evolutionary Computation, Vancouver, Jul 24-29, 2016. Piscataway: IEEE, 2016: 2724-2728.
[24]	SHISHIRA S R, KANDASAMY A, CHANDRASEKARAN K. Workload characterization: survey of current approaches and research challenges[C]// Proceedings of the 7th International Conference on Computer and Communication Technology, Allahabad, Nov 24-26, 2017. New York: ACM, 2017: 151-156.
[25]	Google trace[EB/OL]. [2020-06-23]. https://github.com/google/cluster-data.
[26]	REISS C, TUMANOV A, GANGER G R, et al. Heterogeneity and dynamicity of clouds at scale: Google trace analysis[C]// Proceedings of the 3rd ACM Symposium on Cloud Computing. New York: ACM, 2012: 7.
[27]	FAN Z W, HUANG P J, HUANG P S, et al. A feature generation framework for Google trace analysis[C]// Proceedings of the 2015 International Conference on Machine Learning and Cybernetics. Piscataway: IEEE, 2015: 229-234.
[28]	LU C Z, YE K J, XU G Y, et al. Imbalance in the cloud: an analysis on Alibaba cluster trace[C]// Proceedings of the 2017 IEEE International Conference on Big Data, Boston, Dec 11-14, 2017. Washington: IEEE Computer Society, 2017: 2884-2892.
[29]	CHENG Y, CHAI Z, ANWAR A, Characterizing co-located datacenter workloads: an Alibaba case study[J]. arXiv: 1808. 02919, 2018.
[30]	LIU Q X, YU Z B. The elasticity and plasticity in semi-containerized co-locating cloud workload: a view from Alibaba trace[C]// Proceedings of the 2018 ACM Symposium on Cloud Computing, Carlsbad, Oct 11-13, 2018. New York: ACM, 2018: 347-360.
[31]	CHEN Y, GANAPATHI A S, GRIFFITH R, et al. Analysis and lessons from a publicly available Google cluster trace: UCB/EECS-2010-95[R]. Berkeley: University of California, 2010.
[32]	ALAM M, SHAKIL K A, SETHI S. Analysis and clustering of workload in Google cluster trace based on resource usage[C]// Proceedings of the 2016 IEEE International Conference on Computational Science and Engineering, and IEEE International Conference on Embedded and Ubiquitous Computing, and 15th International Symposium on Distributed Computing and Applications for Business Engineering, Paris, Aug 24-26, 2016. Washington: IEEE Computer Society, 2016: 740-747.
[33]	CHEN W Y, YE K J, WANG Y, et al. How does the workload look like in production cloud? Analysis and clustering of workloads on Alibaba cluster trace[C]// Proceedings of the 24th IEEE International Conference on Parallel and Distributed Systems, Singapore, Dec 11-13, 2018. Piscataway: IEEE, 2018: 102-109.

Workload Characterization of Online and Offline Services in Co-located Data Centers

混部数据中心在线离线服务特征分析

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 29

References 33

Related Articles 15

Recommended Articles

Metrics

[1]	YU Daming, ZHANG Zhen. FSDC: Flexible and Highly Scalable Data Center Network Structure [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(4): 855-864.
[2]	GUAN Zheng, HU Yang, YANG Zhijun, HE Min. Weighted Scheduling Algorithm Based on In-Band Full-Duplex Link for Distributed WLAN [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(2): 372-383.
[3]	LI Chengyan, SONG Yue, MA Jintao. RIOPSO Algorithm for Fuzzy Cloud Resource Scheduling Problem [J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(8): 1534-1545.
[4]	YE Jin, XIE Ziqi, XIAO Qingyu, SONG Ling, LI Xiaohuan. Inferring Coflow Size Mechanism Based on ELM in Data Center Network [J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(2): 261-269.
[5]	GUO Yuhan, YI Peng. Distributed Hybrid Variable Neighborhood Search Algorithm for Carpooling Problem [J]. Journal of Frontiers of Computer Science and Technology, 2019, 13(2): 330-341.
[6]	SUN Huaiying, YU Huiqun, FAN Guisheng, CHEN Liqiong. Time Minimized Task Scheduling in Hadoop with SDN [J]. Journal of Frontiers of Computer Science and Technology, 2018, 12(11): 1767-1776.
[7]	PEI Shujun, SONG Dongmei, KONG Dekai. Application of Fast Pruning Algorithm in Map/Reduce for Complex Tasks Scheduling [J]. Journal of Frontiers of Computer Science and Technology, 2018, 12(1): 72-81.
[8]	YU Yajun, LIU Zheng, XU Mingwei. Research on TCP Incast in Data Center Networks [J]. Journal of Frontiers of Computer Science and Technology, 2017, 11(9): 1361-1378.
[9]	LI Fangfang, LIU Chong, YU Ge. Scheduling Algorithm of Events with Imprecise Timestamps for CPS [J]. Journal of Frontiers of Computer Science and Technology, 2017, 11(6): 887-896.
[10]	ZHANG Yiwen, WANG Cheng. Reliability-Aware Energy Management Scheduling Algorithm for Periodic Task [J]. Journal of Frontiers of Computer Science and Technology, 2017, 11(5): 833-841.
[11]	ZHANG Feipeng, CHEN Lin, ZHANG Jingjing. Measurement and Performance Bottleneck Analysis Method for Large-Scale Complex Networks [J]. Journal of Frontiers of Computer Science and Technology, 2017, 11(2): 262-270.
[12]	WANG Desheng, ZHANG Weizhe, HAO Meng, LU Gangzhao, BAI Enci. Research of Adaptive Virtual Machine Memory Scheduling Algorithm in Cloud Computing Environment [J]. Journal of Frontiers of Computer Science and Technology, 2017, 11(1): 70-79.
[13]	ZHU Peng, HE Kun, CAO Weigang, YANG Huan. Caving-Degree Based Greedy Scheduling Algorithm for Three-Dimensional Space-Time Optimization Problem [J]. Journal of Frontiers of Computer Science and Technology, 2016, 10(8): 1051-1062.
[14]	GAO Renfei, WU Jigang, ZHOU Ying, ZHANG Yaoguo. Virtual Machine Assignment for Minimizing Data Latency [J]. Journal of Frontiers of Computer Science and Technology, 2016, 10(7): 924-935.
[15]	ZHOU Dan, GE Hongwei, SU Shuzhi, YUAN Yunhao. Particle Compaction and Scheduling Based Particle Swarm Optimization [J]. Journal of Frontiers of Computer Science and Technology, 2016, 10(5): 742-750.