In physics, the word entropy has important physical implications as the amount of "disorder" of a system. In mathematics, a more abstract definition is used. The (Shannon) entropy of a variable X is defined as


bits, where P(x) is the probability that X is in the state x, and Plog_2P is defined as 0 if P=0. The joint entropy of variables X_1, ..., X_n is then defined by


See also

Differential Entropy, Information Theory, Kolmogorov Entropy, Maximum Entropy Method, Metric Entropy, Mutual Information, Nat, Ornstein's Theorem, Redundancy, Relative Entropy, Topological Entropy

Explore with Wolfram|Alpha


Ellis, R. S. Entropy, Large Deviations, and Statistical Mechanics. New York: Springer-Verlag, 1985.Havil, J. "A Measure of Uncertainty." §14.1 in Gamma: Exploring Euler's Constant. Princeton, NJ: Princeton University Press, pp. 139-145, 2003.Khinchin, A. I. Mathematical Foundations of Information Theory. New York: Dover, 1957.Lasota, A. and Mackey, M. C. Chaos, Fractals, and Noise: Stochastic Aspects of Dynamics, 2nd ed. New York: Springer-Verlag, 1994.Ott, E. "Entropies." §4.5 in Chaos in Dynamical Systems. New York: Cambridge University Press, pp. 138-144, 1993.Rothstein, J. "Information, Measurement, and Quantum Mechanics." Science 114, 171-175, 1951.Schnakenberg, J. "Network Theory of Microscopic and Macroscopic Behavior of Master Equation Systems." Rev. Mod. Phys. 48, 571-585, 1976.Shannon, C. E. "A Mathematical Theory of Communication." The Bell System Technical J. 27, 379-423 and 623-656, July and Oct. 1948., C. E. and Weaver, W. Mathematical Theory of Communication. Urbana, IL: University of Illinois Press, 1963.

Referenced on Wolfram|Alpha


Cite this as:

Weisstein, Eric W. "Entropy." From MathWorld--A Wolfram Web Resource.

Subject classifications