Automatic differentiation is the evaluation of derivatives of a computation by applying the chain rule to the elementary operations that the computation performs (Baydin et al. 2018). It uses local derivatives of the operations at the computed intermediate values.
In forward mode, let be an intermediate operation in a computation whose
input is
. The propagation rule for
is
More generally, initializing the input with a direction gives the product
,
where
is the Jacobian matrix of the computation.
In reverse mode with scalar output ,
initialize the output sensitivity
to 1 and all other sensitivities to 0. Process the operations
backward, adding
to
and
to
for each operation
.
Additions accumulate contributions when an intermediate value is reused. At the inputs,
the sensitivities give the gradient of
.