Permutation Test

A permutation test is a hypothesis testing procedure in which the observed test statistic is compared with its distribution under transformations of the data that leave the null hypothesis invariant. Let be a finite set of allowed permutations, let denote the observed data, and let be a statistic for which larger values are more extreme. If the null hypothesis makes the transformed datasets equally likely, the exact one-sided -value is

where is the indicator function of an event . A two-sided test instead uses a prespecified two-sided measure of extremeness.

When enumerating all allowed permutations is impractical, transformations , ..., can be sampled. A standard Monte Carlo estimate that avoids reporting a zero -value is

The test is valid when its null hypothesis makes the transformed datasets equally likely and the transformations respect the experimental design. Arbitrarily shuffling observations for which this condition fails does not give an exact test.