TOPICS
Search

Central Limit Theorem

The central limit theorem states that the normalized sum of independent random variates with finite variances approaches a normal distribution. Let X_1,X_2,...,X_N be a set of N independent random variates and each X_i have an arbitrary probability distribution P(x_1,...,x_N) with mean mu_i and a finite variance sigma_i^2. Then the normal form variate

X_(norm)=(sum_(i=1)^(N)x_i-sum_(i=1)^(N)mu_i)/(sqrt(sum_(i=1)^(N)sigma_i^2))
(1)

has a limiting cumulative distribution function which approaches a normal distribution.

Under additional conditions on the distribution of the addend, the probability density itself is also normal (Feller 1971) with mean mu=0 and variance sigma^2=1. If conversion to normal form is not performed, then the variate

X=1/Nsum_(i=1)^Nx_i
(2)

is normally distributed with mu_X=mu_x and sigma_X=sigma_x/sqrt(N).

Kallenberg (1997) gives a six-line proof of the central limit theorem. For an elementary, but slightly more cumbersome proof of the central limit theorem, consider the inverse Fourier transform of P_X(f).

F_f^(-1)[P_X(f)](x)=int_(-infty)^inftye^(2piifX)P(X)dX
(3)
=int_(-infty)^inftysum_(n=0)^(infty)((2piifX)^n)/(n!)P(X)dX
(4)
=sum_(n=0)^(infty)((2piif)^n)/(n!)int_(-infty)^inftyX^nP(X)dX
(5)
=sum_(n=0)^(infty)((2piif)^n)/(n!)<X^n>.
(6)

Now write

<X^n>=<N^(-n)(x_1+x_2+...+x_N)^n> =int_(-infty)^inftyN^(-n)(x_1+...+x_N)^nP(x_1)...P(x_N)dx_1...dx_N,
(7)

so we have

F_f^(-1)[P_X(f)](x)=sum_(n=0)^(infty)((2piif)^n)/(n!)<X^n>
(8)
=sum_(n=0)^(infty)((2piif)^n)/(n!)int_(-infty)^inftyN^(-n)(x_1+...+x_N)^n×P(x_1)...P(x_N)dx_1...dx_N
(9)
=int_(-infty)^inftysum_(n=0)^(infty)[(2piif(x_1+...+x_N))/N]^n1/(n!)P(x_1)...P(x_N)dx_1...dx_N
(10)
=int_(-infty)^inftye^(2piif(x_1+...+x_N)/N)P(x_1)...P(x_N)dx_1...dx_N
(11)
=[int_(-infty)^inftye^(2piifx_1/N)P(x_1)dx_1]×...×[int_(-infty)^inftye^(2piifx_N/N)P(x_N)dx_N]
(12)
=[int_(-infty)^inftye^(2piifx/N)P(x)dx]^N
(13)
={int_(-infty)^infty[1+((2piif)/N)x+1/2((2piif)/N)^2x^2+...]P(x)dx}^N
(14)
=[1+(2piif)/N<x>-((2pif)^2)/(2N^2)<x^2>+O(N^(-3))]^N
(15)
=exp{Nln[1+(2piif)/N<x>-((2pif)^2)/(2N^2)<x^2>+O(N^(-3))]}.
(16)

Now expand

ln(1+x)=x-1/2x^2+1/3x^3+...,
(17)

so

F_f^(-1)[P_X(f)](x) approx exp{N[(2piif)/N<x>-((2pif)^2)/(2N^2)<x^2>+1/2((2piif)^2)/(N^2)<x>^2+O(N^(-3))]}
(18)
=exp[2piif<x>-((2pif)^2(<x^2>-<x>^2))/(2N)+O(N^(-2))]
(19)
approx exp[2piifmu_x-((2pif)^2sigma_x^2)/(2N)],
(20)

since

mu_x=<x>
(21)
sigma_x^2=<x^2>-<x>^2.
(22)

Taking the Fourier transform,

P_X=int_(-infty)^inftye^(-2piifx)F^(-1)[P_X(f)]df
(23)
=int_(-infty)^inftye^(2piif(mu_x-x)-(2pif)^2sigma_x^2/2N)df.
(24)

This is of the form

int_(-infty)^inftye^(iaf-bf^2)df,
(25)

where a=2pi(mu_x-x) and b=(2pisigma_x)^2/2N. But this is a Fourier transform of a Gaussian function, so

int_(-infty)^inftye^(iaf-bf^2)df=e^(-a^2/4b)sqrt(pi/b)
(26)

(e.g., Abramowitz and Stegun 1972, p. 302, equation 7.4.6). Therefore,

P_X=sqrt(pi/(((2pisigma_x)^2)/(2N)))exp{(-[2pi(mu_x-x)]^2)/(4((2pisigma_x)^2)/(2N))}
(27)
=sqrt((2piN)/(4pi^2sigma_x^2))exp[-(4pi^2(mu_x-x)^22N)/(4·4pi^2sigma_x^2)]
(28)
=(sqrt(N))/(sigma_xsqrt(2pi))e^(-(mu_x-x)^2N/2sigma_x^2).
(29)

But sigma_X=sigma_x/sqrt(N) and mu_X=mu_x, so

P_X=1/(sigma_Xsqrt(2pi))e^(-(mu_X-x)^2/2sigma_X^2).
(30)

The "fuzzy" central limit theorem says that data which are influenced by many small and unrelated random effects are approximately normally distributed.

The fact that widely different underlying statistical distributions have the same limiting normal form is an instance of the Lindeberg universality principle and a basic example of universality in probability. As a result, details of the microscopic distributions can be forgotten while a common macroscopic law remains (Tropp 2023, pp. 275-276).

See also

Berry-Esséen Theorem, Fourier Transform--Gaussian, Lindeberg Condition, Lindeberg-Feller Central Limit Theorem, Lindeberg Universality Principle, Lyapunov Condition, Universality Explore this topic in the MathWorld classroom

Explore with Wolfram|Alpha

References

Abramowitz, M. and Stegun, I. A. (Eds.). Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables, 9th printing. New York: Dover, 1972.Feller, W. "The Fundamental Limit Theorems in Probability." Bull. Amer. Math. Soc. 51, 800-832, 1945.Feller, W. An Introduction to Probability Theory and Its Applications, Vol. 1, 3rd ed. New York: Wiley, p. 229, 1968.Feller, W. An Introduction to Probability Theory and Its Applications, Vol. 2, 3rd ed. New York: Wiley, 1971.Kallenberg, O. Foundations of Modern Probability. New York: Springer-Verlag, 1997.Lindeberg, J. W. "Eine neue Herleitung des Exponentialgesetzes in der Wahrscheinlichkeitsrechnung." Math. Z. 15, 211-225, 1922.Spiegel, M. R. Theory and Problems of Probability and Statistics. New York: McGraw-Hill, pp. 112-113, 1992.Tropp, J. A. Probability Theory & Computational Mathematics. Caltech CMS Lecture Notes 2023-01. Pasadena, CA: California Institute of Technology, pp. 275-276, 2023. https://doi.org/10.7907/q75sz-e1e79.Trotter, H. F. "An Elementary Proof of the Central Limit Theorem." Arch. Math. 10, 226-234, 1959.Zabell, S. L. "Alan Turing and the Central Limit Theorem." Amer. Math. Monthly 102, 483-494, 1995.

Referenced on Wolfram|Alpha

Central Limit Theorem

Cite this as:

Weisstein, Eric W. "Central Limit Theorem." From MathWorld--A Wolfram Resource. https://mathworld.wolfram.com/CentralLimitTheorem.html

Subject classifications