Principal Component Analysis

Principal component analysis (PCA) is a method for replacing a collection of possibly correlated variables by uncorrelated linear combinations called principal components. Each coefficient vector is a unit vector. The components are ordered so that the first accounts for the greatest possible variance, the second accounts for the greatest remaining variance subject to being orthogonal to the first, and so on.

Let and , and let the rows of the real matrix be observations centered by subtracting each column mean. Put

(1)

The matrix is the sample covariance matrix. If its eigenvectors form an orthonormal set and its eigenvalues are ordered so that

			(2)
			(3)

then the th principal-component score of the th observation is

(4)

The variance of the th component is , and, when the total variance is positive, the fraction that it explains is

(5)

For an integer with , keeping only the first components gives a lower-dimensional representation that maximizes retained variance and, equivalently, minimizes the sum of squared orthogonal reconstruction errors among -dimensional linear subspaces. If each variable has positive sample variance, PCA can also be performed on standardized variables using their correlation matrix instead of their covariance matrix. The method originated with Pearson's closest-fitting subspaces and was developed and named by Hotelling (Pearson 1901, Hotelling 1933).