The entropy chain rule expresses the joint entropy of random variables as a sum
of conditional entropies. For discrete random variables 
with finite alphabets and logarithm
base 
,
 |
where the first term is 
. The identity
follows from the probability chain rule
by taking the negative expectation value of
the logarithm of the joint probability
mass function (Gray 1990, p. 41).
There is also a chain rule for relative entropy. For joint probability mass functions 
and 
on finite alphabets, with 
wherever 
,
 |
where the marginal distributions are denoted by 
and 
, and the conditional
distributions given 
by 
and 
. Terms with 
are omitted. Related decompositions for mutual
information follow by applying the entropy chain rule to 
(Cover and Thomas 2006, pp. 21-25).
