The entropy chain rule expresses the joint entropy of random variables as a sum
of conditional entropies. For discrete random variables 
with finite alphabets and logarithm
base 
,
 |
where the first term is 
. The identity follows
from the probability chain rule by taking
the negative expectation value of the logarithm
of the joint probability mass function
(Gray 1990, p. 41).
There is also a chain rule for relative entropy. For joint probability mass functions 
and 
on finite alphabets, with 
wherever 
,
 |
where the marginal distributions are denoted by 
and 
,
and the conditional distributions given
by 
and 
.
Terms with 
are omitted. Related decompositions for mutual
information follow by applying the entropy chain rule to 
(Cover and Thomas 2006, pp. 21-25).
