TOPICS
Search

Joint Entropy

The joint entropy of discrete random variables X_1,...,X_n is the Shannon entropy of their joint probability distribution,

H_b(X_1,...,X_n)=-sum_(x_1,...,x_n)p(x_1,...,x_n)log_bp(x_1,...,x_n).

For two random variables, this gives

H_b(X,Y)=H_b(X)+H_b(Y|X).

In particular, H_b(X,Y)<=H_b(X)+H_b(Y), with equality precisely when X and Y are independent.

See also

Conditional Entropy, Mutual Information, Shannon Entropy

Explore with Wolfram|Alpha

References

Cover, T. M. and Thomas, J. A. Elements of Information Theory. New York: Wiley, 1991.Shannon, C. E. "A Mathematical Theory of Communication." Bell System Technical J. 27, 379-423, 1948. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x.

Cite this as:

Weisstein, Eric W. "Joint Entropy." From MathWorld--A Wolfram Resource. https://mathworld.wolfram.com/JointEntropy.html

Subject classifications