The relative entropy of two discrete distributions 
and 
, also called the Kullback-Leibler divergence, is
 |
where 
,
terms with 
are taken to be 0, and the value is infinite if 
and 
for some 
. Relative entropy is nonnegative
and vanishes precisely when 
. It is jointly convex in 
, but is neither symmetric nor a
metric.
