Stochastic approximation is a family of iterative methods for finding a root or optimum when observations are corrupted by noise. A typical recursion has the form
 |
where 
is a decreasing step size, 
is the mean update, and 
is a noise term. Standard convergence hypotheses
include 
and 
together with stability and suitable control of the noise.
The Robbins-Monro stochastic approximation estimates a root of a regression function, while stochastic gradient methods use noisy estimates of the gradient of an objective function.
