TOPICS
Search

Automatic Differentiation


Automatic differentiation is the evaluation of derivatives of a computation by applying the chain rule to the elementary operations that the computation performs (Baydin et al. 2018). It uses local derivatives of the operations at the computed intermediate values.

In forward mode, let z=phi(u,v) be an intermediate operation in a computation whose input is x. The propagation rule for z^.=dz/dx is

 z^.=phi_u(u,v)u^.+phi_v(u,v)v^..

More generally, initializing the input with a direction p gives the product Jp, where J is the Jacobian matrix of the computation.

In reverse mode with scalar output y, initialize the output sensitivity y^_ to 1 and all other sensitivities to 0. Process the operations backward, adding z^_phi_u(u,v) to u^_ and z^_phi_v(u,v) to v^_ for each operation z=phi(u,v). Additions accumulate contributions when an intermediate value is reused. At the inputs, the sensitivities give the gradient of y.


See also

Chain Rule, Differentiation, Numerical Differentiation

Explore with Wolfram|Alpha

References

Baydin, A. G.; Pearlmutter, B. A.; Radul, A. A.; and Siskind, J. M. "Automatic Differentiation in Machine Learning: A Survey." J. Mach. Learn. Res. 18, 1-43, 2018. https://www.jmlr.org/papers/v18/17-468.html.

Cite this as:

Weisstein, Eric W. "Automatic Differentiation." From MathWorld--A Wolfram Resource. https://mathworld.wolfram.com/AutomaticDifferentiation.html

Subject classifications