TOPICS
Search

Conditional Entropy

The conditional entropy of a discrete random variable Y given a discrete random variable X, for b>1, is

H_b(Y|X)=-sum_(x,y)p(x,y)log_bp(y|x),

where terms with p(x,y)=0 are taken to be 0. When the entropies on the right are finite, it also satisfies

H_b(Y|X)=H_b(X,Y)-H_b(X).

Conditional entropy satisfies 0<=H_b(Y|X)<=H_b(Y). The upper bound is attained when X and Y are independent.

See also

Conditional Mutual Information, Entropy Chain Rule, Joint Entropy, Mutual Information, Shannon Entropy

Explore with Wolfram|Alpha

References

Cover, T. M. and Thomas, J. A. Elements of Information Theory. New York: Wiley, 1991.Shannon, C. E. "A Mathematical Theory of Communication." Bell System Technical J. 27, 379-423, 1948. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x.

Cite this as:

Weisstein, Eric W. "Conditional Entropy." From MathWorld--A Wolfram Resource. https://mathworld.wolfram.com/ConditionalEntropy.html

Subject classifications