Robbins-Monro stochastic approximation is a stochastic approximation method that places conditions on iterative step sizes and converges under mild conditions. However, the method requires knowledge of the analytical gradient of the function under consideration.
Kiefer and Wolfowitz (1952) developed a finite difference version of the Robbins-Monro method which maintains the nice convergence properties, while obviating the need for knowledge of the analytic form of the gradient.
Kiefer, J. and Wolfowitz, J. "Stochastic Estimation of the Maximum of a Regression Function." Ann. Math. Stat.23,
462-466, 1952.Robbins, H. and Munro, S. "A Stochastic Approximation
Method." Ann. Math. Stat.22, 400-407, 1951.
