Stochastic modified equations for the asynchronous stochastic gradient descent

Journal Article (Journal Article)

We propose stochastic modified equations (SMEs) for modelling the asynchronous stochastic gradient descent (ASGD) algorithms. The resulting SME of Langevin type extracts more information about the ASGD dynamics and elucidates the relationship between different types of stochastic gradient algorithms. We show the convergence of ASGD to the SME in the continuous time limit, as well as the SME's precise prediction to the trajectories of ASGD with various forcing terms. As an application, we propose an optimal mini-batching strategy for ASGD via solving the optimal control problem of the associated SME.

Full Text

Duke Authors

Cited Authors

  • An, J; Lu, J; Ying, L

Published Date

  • January 1, 2020

Published In

Volume / Issue

  • 9 / 4

Start / End Page

  • 851 - 873

Electronic International Standard Serial Number (EISSN)

  • 2049-8772

Digital Object Identifier (DOI)

  • 10.1093/IMAIAI/IAZ030

Citation Source

  • Scopus