Covariance provides a measure of the strength of the correlation between two or more sets of random variates. The covariance for two random variates 
and 
,
each with sample size 
, is defined by the expectation
value
 |  |  |
(1)
|
 |  |  |
(2)
|
where 
and 
are the respective means,
which can be written out explicitly as
 |
(3)
|
For uncorrelated variates,
 |
(4)
|
so the covariance is zero. However, if the variables are correlated in some way, then their covariance will be nonzero. In fact, if 
, then 
tends to increase as 
increases, and if 
, then 
tends to decrease as 
increases. Note that while statistically independent variables
are always uncorrelated, the converse is not necessarily true.
In the special case of 
,
 |  |  |
(5)
|
 |  |  |
(6)
|
so the covariance reduces to the usual variance 
. This motivates the use
of the symbol 
,
which then provides a consistent way of denoting the variance
as 
, where 
is the standard deviation.
The derived quantity
 |  |  |
(7)
|
 |  |  |
(8)
|
is called statistical correlation of 
and 
.
The covariance is especially useful when looking at the variance of the sum of two random variates, since
 |
(9)
|
The covariance is symmetric by definition since
 |
(10)
|
Given 
random variates denoted 
,
..., 
, the covariance 
of 
and 
is defined by
 |  |  |
(11)
|
 |  |  |
(12)
|
where 
and 
are the means
of 
and 
, respectively. The matrix 
of the quantities 
is called the covariance
matrix.
The covariance obeys the identities
 |  |  |
(13)
|
 |  |  |
(14)
|
 |  |  |
(15)
|
 |  |  |
(16)
|
By induction, it therefore follows that
 |  |  |
(17)
|
 |  |  |
(18)
|
 |  |  |
(19)
|
 |  |  |
(20)
|
 |  |  |
(21)
|
