The cross entropy of discrete probability distributions 
and 
on the same countable set 
is
 |
where a term is infinite when 
and 
. In terms of the Shannon
entropy 
and relative entropy 
,
 |
It therefore equals 
exactly when 
.
Cross Entropy
See also
Relative Entropy, Shannon EntropyExplore with Wolfram|Alpha
References
Cover, T. M. and Thomas, J. A. Elements of Information Theory. New York: Wiley, 1991.Kullback, S. and Leibler, R. A. "On Information and Sufficiency." Ann. Math. Stat. 22, 79-86, 1951. https://doi.org/10.1214/aoms/1177729694.Cite this as:
Weisstein, Eric W. "Cross Entropy." From MathWorld--A Wolfram Resource. https://mathworld.wolfram.com/CrossEntropy.html