The Shannon entropy of a discrete random variable countable
set
where bits , while nats .
For a countably infinite sum may be infinite.
Shannon entropy is nonnegative . If lower
bound is attained when upper
bound is attained when uniform distribution
on
The joint Shannon entropy of discrete random variables
See also Conditional Entropy ,
Cross Entropy ,
Differential Entropy ,
Entropy ,
Information Theory ,
Joint
Entropy ,
Max-Entropy ,
Min-Entropy ,
Mutual Information ,
Nat ,
Relative Entropy ,
Rényi
Entropy ,
Tsallis Entropy ,
von
Neumann Entropy
Explore with Wolfram|Alpha
References Cover, T. M. and Thomas, J. A. Elements of Information Theory. Havil, J. "A
Measure of Uncertainty." §14.1 in Gamma:
Exploring Euler's Constant. Khinchin, A. I. Mathematical
Foundations of Information Theory. Shannon,
C. E. "A Mathematical Theory of Communication." Bell System Technical
J. 27 , 379-423, 1948. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x . Shannon,
C. E. and Weaver, W. The
Mathematical Theory of Communication. Referenced on Wolfram|Alpha Shannon Entropy
Cite this as:
Weisstein, Eric W. "Shannon Entropy."
From MathWorld https://mathworld.wolfram.com/ShannonEntropy.html
Subject classifications
taking values in a countable
set 
,
with probability mass function 
, is
and 
is defined to be 0 (Shannon 1948). Taking 
gives entropy in bits, while 
gives entropy in nats.
For a countably infinite 
, the sum may be infinite.
has 
elements, then 
. The lower
bound is attained when 
takes a single value with probability 1, and the upper
bound is attained when 
has the uniform distribution
on 
.
, ..., 
, with joint probability mass function 
, is
