关键词: 机器学习, 优化, 随机梯度下降（SGD）, 随机方差消减梯度法（SVRG）, 分布式计算

Abstract:

Machine learning problems are usually converted into an objective function to optimize, and the optimization algorithms are the important tool to solve the parameters in the objective function. Currently, stochastic gradient descent (SGD) is one of the most widely used optimization methods, but it is susceptible to noise gradient so as to get sub-linear convergence rate. The improved stochastic variance reduction gradient (SVRG) can achieve linear convergence rate, but SVRG is a serial algorithm. In order to deal with the distributed training problem of large-scale data set, this paper designs topkSVRG based on SVRG algorithm. The improvement is that the master node maintains a global model, and the local nodes update the local model according to the local data. In each epoch, the global model is updated by selecting k local models which have the smallest distance from the current global model. Usually with a bigger k, the model can converge faster, while with a smaller k, the convergence rate can be guar-   anteed. topkSVRG has been proved linear convergence rate by theoretical analysis. topkSVRG is implemented on Spark, and experiments demonstrate its efficiency compared with Mini-Batch SGD, CoCoA, Splash, and so on.

Key words: machine learning, optimization, stochastic gradient descent (SGD), stochastic variance reduction gradient (SVRG), distributed computing