Exploring foci of:
arXiv (Cornell University)
Asymptotics of Stochastic Gradient Descent with Dropout Regularization in Linear Models
September 2024 • J. Jenny Li, Johannes Schmidt-Hieber, Wei Biao Wu
This paper proposes an asymptotic theory for online inference of the stochastic gradient descent (SGD) iterates with dropout regularization in linear regression. Specifically, we establish the geometric-moment contraction (GMC) for constant step-size SGD dropout iterates to show the existence of a unique stationary distribution of the dropout recursive function. By the GMC property, we provide quenched central limit theorems (CLT) for the difference between dropout and $\ell^2$-regularized iterates, regardless of …
Stochastic Gradient Descent
Mathematics
Gradient Descent
Computer Science
Artificial Intelligence
Machine Learning