Nonparametric tests

OneSampleADTest(x::AbstractVector{<:Real}, d::UnivariateDistribution)

Perform a one-sample Anderson–Darling test of the null hypothesis that the data in vector x come from the distribution d against the alternative hypothesis that the sample is not drawn from d.

Implements: pvalue

KSampleADTest(xs::AbstractVector{<:Real}...; modified = true, nsim = 0)

Perform a $k$-sample Anderson–Darling test of the null hypothesis that the data in the $k$ vectors xs come from the same distribution against the alternative hypothesis that the samples come from different distributions.

modified parameter enables a modified test calculation for samples whose observations do not all coincide.

If nsim is equal to 0 (the default) the asymptotic calculation of p-value is used. If it is greater than 0, an estimation of p-values is used by generating nsim random splits of the pooled data on $k$ samples, evaluating the AD statistics for each split, and computing the proportion of simulated values which are greater or equal to observed. This proportion is reported as p-value estimate.

Implements: pvalue

References

• Scholz and Stephens (1987). K-Sample Anderson-Darling Tests

BinomialTest(x::Integer, n::Integer, p::Real = 0.5)
BinomialTest(x::AbstractVector{Bool}, p::Real = 0.5)

Perform a binomial test of the null hypothesis that the distribution from which x successes were encountered in n draws (or alternatively from which the vector x was drawn) has success probability p against the alternative hypothesis that the success probability is not equal to p.

Computed confidence intervals by default are Clopper-Pearson intervals. See the confint(::BinomialTest) documentation for a list of supported methods to compute confidence intervals.

Implements: pvalue, confint(::BinomialTest)

confint(test::BinomialTest; level = 0.95, tail = :both, method = :clopper_pearson)

Compute a confidence interval with coverage level for a binomial proportion using one of the following methods. Possible values for method are:

• :clopper_pearson (default): Clopper-Pearson interval is based on the binomial distribution. The empirical coverage is never less than the nominal coverage of level; it is usually too conservative.
• :wald: Wald (or normal approximation) interval relies on the standard approximation of the actual binomial distribution by a normal distribution. Coverage can be erratically poor for success probabilities close to zero or one.
• :waldcc: Wald interval with a continuity correction that extends the interval by 1/2n on both ends.
• :wilson: Wilson score interval relies on a normal approximation. In contrast to :wald, the standard deviation is not approximated by an empirical estimate, resulting in good empirical coverages even for small numbers of draws and extreme success probabilities.
• :jeffrey: Jeffreys interval is a Bayesian credible interval obtained by using a non-informative Jeffreys prior. The interval is very similar to the Wilson interval.
• :agresti_coull: Agresti-Coull interval is a simplified version of the Wilson interval; both are centered around the same value. The Agresti Coull interval has higher or equal coverage.
• :arcsine: Confidence interval computed using the arcsine transformation to make $var(p)$ independent of the probability $p$.

References

• Brown, Cai, and DasGupta (2001). Interval estimation for a binomial proportion
• Pires and Amado (2008). Interval Estimators for a Binomial Proportion: Comparison of Twenty Methods

External links

• Binomial confidence interval on Wikipedia

FisherExactTest(a::Integer, b::Integer, c::Integer, d::Integer)

Perform Fisher's exact test of the null hypothesis that the success probabilities $a/c$ and $b/d$ are equal, that is the odds ratio $(a/c) / (b/d)$ is one, against the alternative hypothesis that they are not equal.

See pvalue(::FisherExactTest) and confint(::FisherExactTest) for details about the computation of the default p-value and confidence interval, respectively.

The contingency table is structured as:

Implements: pvalue(::FisherExactTest), confint(::FisherExactTest)

References

• Fay (2010). Confidence Intervals That Match Fisher's Exact or Blaker's Exact Tests

confint(x::FisherExactTest; level::Float64=0.95, tail=:both, method=:central)

Compute a confidence interval with coverage level. One-sided intervals are based on Fisher's non-central hypergeometric distribution. For tail = :both, the only method implemented yet is the central interval (:central).

References

• Gibbons and Pratt (1975). P-Values: Interpretation and Methodology
• Fay (2010). Confidence Intervals That Match Fisher's Exact or Blaker's Exact Tests

pvalue(x::FisherExactTest; tail = :both, method = :central)

Compute the p-value for a given Fisher exact test.

The one-sided p-values are based on Fisher's non-central hypergeometric distribution $f_ω(i)$ with odds ratio $ω$:

\[ \begin{align*} p_ω^{(\text{left})} &=\sum_{i ≤ a} f_ω(i)\\ p_ω^{(\text{right})} &=\sum_{i ≥ a} f_ω(i) \end{align*}\]

For tail = :both, possible values for method are:

• :central (default): Central interval, i.e. the p-value is two times the minimum of the one-sided p-values.
• :minlike: Minimum likelihood interval, i.e. the p-value is computed by summing all tables with the same marginals that are equally or less probable:

  \[ p_ω = \sum_{f_ω(i)≤ f_ω(a)} f_ω(i)\]

References

• Gibbons and Pratt (1975). P-Values: Interpretation and Methodology
• Fay (2010). Confidence Intervals That Match Fisher's Exact or Blaker's Exact Tests

ExactOneSampleKSTest(x::AbstractVector{<:Real}, d::UnivariateDistribution)

Perform a one-sample exact Kolmogorov–Smirnov test of the null hypothesis that the data in vector x comes from the distribution d against the alternative hypothesis that the sample is not drawn from d.

Implements: pvalue

ApproximateOneSampleKSTest(x::AbstractVector{<:Real}, d::UnivariateDistribution)

Perform an asymptotic one-sample Kolmogorov–Smirnov test of the null hypothesis that the data in vector x comes from the distribution d against the alternative hypothesis that the sample is not drawn from d.

Implements: pvalue

ApproximateTwoSampleKSTest(x::AbstractVector{<:Real}, y::AbstractVector{<:Real})

Perform an asymptotic two-sample Kolmogorov–Smirnov-test of the null hypothesis that x and y are drawn from the same distribution against the alternative hypothesis that they come from different distributions.

Implements: pvalue

External links

• Approximation of one-sided test (Encyclopedia of Mathematics)

KruskalWallisTest(groups::AbstractVector{<:Real}...)

Perform Kruskal-Wallis rank sum test of the null hypothesis that the groups $\mathcal{G}$ come from the same distribution against the alternative hypothesis that at least one group stochastically dominates one other group.

The Kruskal-Wallis test is an extension of the Mann-Whitney U test to more than two groups.

The p-value is computed using a $χ^2$ approximation to the distribution of the test statistic $H_c=\frac{H}{C}$:

\[ \begin{align*} H & = \frac{12}{n(n+1)} \sum_{g ∈ \mathcal{G}} \frac{R_g^2}{n_g} - 3(n+1)\\ C & = 1-\frac{1}{n^3-n}\sum_{t ∈ \mathcal{T}} (t^3-t), \end{align*}\]

where $\mathcal{T}$ is the set of the counts of tied values at each tied position, $n$ is the total number of observations across all groups, and $n_g$ and $R_g$ are the number of observations and the rank sum in group $g$, respectively. See references for further details.

Implements: pvalue

References

• Meyer and Seaman (2008). Expanded Table of the Kruskal-Wallis Statistic
• Meyer and Seaman (2013). A Comparison of the Exact Kruskal-Wallis Distribution to Asymptotic Approximations for All Sample Sizes up to 105

External links

• Kruskal-Wallis test on Wikipedia

MannWhitneyUTest(x::AbstractVector{<:Real}, y::AbstractVector{<:Real}; method = :auto)

Perform a Mann-Whitney U test of the null hypothesis that x and y are drawn from the same distribution, against the alternative that one tends to exceed the other.

Equality of the two distributions is what makes the test exact: it renders the pooled observations exchangeable, which is what the null distribution of the statistic rests on. This is not a test of P(x > y) = P(y > x) alone. That weaker equality permits the two distributions to differ in spread, and then the test does not hold its level: for Normal(0, 1) against Normal(0, 3) (σ = 3, so three times the spread), where it holds exactly, a nominal 0.05 test rejects about 13% of the time at nx, ny = 30, 10, and about 1.7% at 10, 30.

Under a shift model the location estimated is the median of x - y, reported by hodgeslehmann.

The Mann-Whitney U test is sometimes known as the Wilcoxon rank-sum test.

MannWhitneyUTest chooses between the exact and the approximate test by the tie pattern and the pooled sample size nx + ny: with no tied ranks it is exact for nx + ny ≤ 50, with tied ranks for nx + ny ≤ 10, and approximate above whichever of those two applies.

method overrides that choice:

• :auto (the default) applies the rule above.
• :exact and :approximate force the corresponding test, which is what an analysis that must reproduce across versions of this package should do.
• a callable is passed (; nx, ny, ties, tie_adjustment) and must return :exact or :approximate.

Equivalently, construct ExactMannWhitneyUTest or ApproximateMannWhitneyUTest directly.

Implements: pvalue, confint, hodgeslehmann

ExactMannWhitneyUTest(x::AbstractVector{<:Real}, y::AbstractVector{<:Real})

Perform an exact Mann-Whitney U test of the null hypothesis that x and y are drawn from the same distribution, against the alternative that one tends to exceed the other. See MannWhitneyUTest on what that null does and does not assume.

When there are no tied ranks, the exact p-value is computed using the wilcoxcdf and wilcoxccdf functions from the StatsFuns package. In the presence of tied ranks, a p-value is computed by exhaustive enumeration of the assignments of the pooled midranks to the smaller sample.

The tied route is bounded by MAX_EXACT_ENUMERATION_N: beyond it this test refuses rather than enumerate indefinitely, and method = :approximate is the way on.

Implements: pvalue, confint, hodgeslehmann

ApproximateMannWhitneyUTest(x::AbstractVector{<:Real}, y::AbstractVector{<:Real})

Perform an approximate Mann-Whitney U test of the null hypothesis that x and y are drawn from the same distribution, against the alternative that one tends to exceed the other. See MannWhitneyUTest on what that null does and does not assume.

The p-value is computed using a normal approximation to the distribution of the Mann-Whitney U statistic, which under the null has mean and variance

\[ \begin{align*} μ_0 & = \frac{n_x n_y}{2}\\ σ^2 & = \frac{n_x n_y}{12}\left(n_x + n_y + 1 - \frac{a}{(n_x + n_y)(n_x + n_y - 1)}\right)\\ a & = \sum_{t \in \mathcal{T}} t^3 - t \end{align*}\]

where $\mathcal{T}$ is the set of the counts of tied values at each tied position. What show reports as normal approximation (μ, σ) is the pair $(U - μ_0, σ)$: the statistic centred at its null mean, and the tie-corrected standard deviation, not $μ_0$ itself.

The confidence interval inverts the same approximation, rather than the exact null distribution.

Implements: pvalue, confint, hodgeslehmann

SignTest(x::AbstractVector{T<:Real}, median::Real = 0)
SignTest(x::AbstractVector{T<:Real}, y::AbstractVector{T<:Real}, median::Real = 0)

Perform a sign test of the null hypothesis that the distribution from which x (or x - y if y is provided) was drawn has median median against the alternative hypothesis that the median is not equal to median.

Implements: pvalue, confint

WaldWolfowitzTest(x::AbstractVector{Bool})
WaldWolfowitzTest(x::AbstractVector{<:Real})

Perform the Wald-Wolfowitz (or Runs) test of the null hypothesis that the given data is random, or independently sampled. The data can come as many-valued or two-valued (Boolean). If many-valued, the sample is transformed by labelling each element as above or below the median.

Implements: pvalue

SignedRankTest(x::AbstractVector{<:Real}; method = :auto)
SignedRankTest(x::AbstractVector{<:Real}, y::AbstractVector{<:Real}; method = :auto)

Perform a Wilcoxon signed rank test of the null hypothesis that the distribution of x (or the difference x - y if y is provided) is symmetric about zero, against the alternative that it is not. Under a location model, where the distribution is symmetric about some θ, that is θ = 0 against θ ≠ 0, with tail = :left and tail = :right giving θ < 0 and θ > 0.

Symmetry is what the test needs, not a zero median: against a null of zero median alone it does not hold its level. The location it estimates is the pseudomedian (see hodgeslehmann), which equals the median when the distribution is symmetric.

SignedRankTest chooses between the exact and the approximate test by the tie pattern and the number n of non-zero observations: with no tied ranks it is exact for n ≤ 50, with tied ranks for n ≤ 15, and approximate above whichever of those two applies.

method overrides that choice:

• :auto (the default) applies the rule above.
• :exact and :approximate force the corresponding test, which is what an analysis that must reproduce across versions of this package should do.
• a callable is passed (; n, n_nonzero, ties, tie_adjustment) and must return :exact or :approximate.

Equivalently, construct ExactSignedRankTest or ApproximateSignedRankTest directly.

Implements: pvalue, confint, hodgeslehmann

ExactSignedRankTest(x::AbstractVector{<:Real}[, y::AbstractVector{<:Real}])

Perform an exact Wilcoxon signed rank test of the null hypothesis that the distribution of x (or the difference x - y if y is provided) is symmetric about zero, against the alternative that it is not. See SignedRankTest on what that null does and does not assume, and on the alternatives the tail keyword selects.

When there are no tied ranks, the exact p-value is computed using the signrankcdf and signrankccdf functions from the StatsFuns package. In the presence of tied ranks, a p-value is computed by exhaustive enumeration of the $2^n$ sign assignments over the non-zero observations.

The tied route is bounded by MAX_EXACT_ENUMERATION_N: beyond it this test refuses rather than enumerate indefinitely, and method = :approximate is the way on.

Implements: pvalue, confint, hodgeslehmann

ApproximateSignedRankTest(x::AbstractVector{<:Real}[, y::AbstractVector{<:Real}])

Perform an approximate Wilcoxon signed rank test of the null hypothesis that the distribution of x (or the difference x - y if y is provided) is symmetric about zero, against the alternative that it is not. See SignedRankTest on what that null does and does not assume.

The p-value is computed using a normal approximation to the distribution of the signed rank statistic $W^+$, which under the null has mean and variance

\[ \begin{align*} μ_0 & = \frac{n(n + 1)}{4}\\ σ^2 & = \frac{n(n + 1)(2n + 1)}{24} - \frac{a}{48}\\ a & = \sum_{t \in \mathcal{T}} t^3 - t \end{align*}\]

where $\mathcal{T}$ is the set of the counts of tied values at each tied position and $n$ counts the non-zero observations. What show reports as normal approximation (μ, σ) is the pair $(W^+ - μ_0, σ)$: the statistic centred at its null mean, and the tie-corrected standard deviation, not $μ_0$ itself.

The confidence interval still inverts the exact null distribution, whichever route the p-value took (see #361).

Implements: pvalue, confint, hodgeslehmann

hodgeslehmann(test)

The Hodges-Lehmann estimator underlying test: the median of the Walsh averages for the signed rank tests, and the median of the cross-group differences for the Mann-Whitney U tests.

This is the point estimate the distribution-free interval returned by confint is built around, and the estimate R's wilcox.test reports where it takes its exact route. On its approximate route R solves for that estimate numerically instead, and reports a slightly different number: 9.71184 against 9.675 on the fifteen-point sample in the tests.

It is generally not the sample median.

Bound on the tied-data enumerations, which is where an exact rank test computes its p-value when the tie total is not zero.

The signed rank test enumerates $2^n$ sign assignments over its $n$ non-zero observations, and refuses past n = MAX_EXACT_ENUMERATION_N. The Mann-Whitney test enumerates $\binom{n_x + n_y}{\min(n_x, n_y)}$ rank assignments, so the bound is on that count rather than on the sample size: it refuses once there are more than 2^MAX_EXACT_ENUMERATION_N of them. Either way the p-value raises rather than run for an unbounded time, and method = :approximate is the way on.

confint does not enumerate, since it inverts the tie-free distribution whichever route the p-value took, so this bound does not reach it. Its own cost is bounded by MAX_EXACT_CI_ESTIMATES. See #7.

At the bound itself a tied signed rank p-value takes about 14 s over its $2^{30}$ sign assignments, and the largest balanced split it admits, 16 against 16, about 20 s over binomial(32, 16) rank assignments. One observation further doubles both.

Largest set of pairwise estimates (Walsh averages, or cross-group differences) that will be materialised. The estimators and intervals below are $O(n^2)$ in memory; beyond this bound they refuse rather than exhaust the machine. Displaying such a test still works: show drops its confidence interval line rather than failing to print at all. See #7 and #97.

The bound is on memory alone, which is what the approximate routes spend: forming this many estimates costs about 8 MB, and sorting them about that again. It admits a signed rank sample of roughly 1400 observations and a two-sample design of 1000 against 1000. The exact routes spend time as well, and are bounded far below this by MAX_EXACT_CI_ESTIMATES.

Largest set of pairwise estimates whose exact interval will be inverted.

Selecting an endpoint from the exact null distribution costs a lattice recursion for every candidate it considers, so this route spends time where the approximate one spends only the memory MAX_PAIRWISE_ESTIMATES bounds, and it needs a bound of its own, far below. Past this one the exact interval is refused; the approximate interval, where the test has one, is not.

Sized by the slower of the two procedures. The signed rank interval scans every candidate, $m/2$ recursions, and at the bound, a sample of 223, takes about 9 s; the two-sample interval bisects instead, about $\log_2(m/2)$ of them, and takes under a second at the same point. The scan grows steeply past it: a signed rank sample of 250 takes 16 s, one of 300 takes 39 s, and one of 400 does not finish.

ComputationTooLarge <: Exception

Thrown where a test refuses a computation whose cost this package bounds, rather than run for an unbounded time or exhaust memory. See MAX_EXACT_ENUMERATION_N, MAX_PAIRWISE_ESTIMATES and MAX_EXACT_CI_ESTIMATES for the three bounds and the way past each.

It is deliberately not an ArgumentError: the argument is not wrong, the cost of answering for it is refused, and show needs to tell that apart from a genuine argument bug in a confint method it knows nothing about.

ExactPermutationTest(x::Vector, y::Vector, f::Function)

Perform a permutation test (a.k.a. randomization test) of the null hypothesis that f(x) is equal to f(y). All possible permutations are sampled.

ApproximatePermutationTest([rng::AbstractRNG,] x::Vector, y::Vector, f::Function, n::Int)

Perform a permutation test (a.k.a. randomization test) of the null hypothesis that f(x) is equal to f(y). n of the factorial(length(x)+length(y)) permutations are sampled at random. A random number generator can optionally be passed as the first argument. The default generator is Random.default_rng().

FlignerKilleenTest(groups)
FlignerKilleenTest(groups::AbstractVector{<:Real}...)

Perform Fligner-Killeen median test of the null hypothesis that the groups have equal variances, a test for homogeneity of variances.

This test is most robust against departures from normality, see references. It is a $k$-sample simple linear rank method that uses the ranks of the absolute values of the centered samples and weights

\[a_{N,i} = \Phi^{-1}(1/2 + (i/2(N+1)))\]

The version implemented here uses median centering in each of the samples.

Implements: pvalue

References

• Conover, Johnson, and Johnson (1981). A Comparative Study of Tests for Homogeneity of Variances, with Applications to the Outer Continental Shelf Bidding Data

External links

• Fligner-Killeen test on Statistical Analysis Handbook

ShapiroWilkTest(X::AbstractVector{<:Real},
                swc::AbstractVector{<:Real}=shapiro_wilk_coefs(length(X));
                sorted::Bool=issorted(X),
                censored::Integer=0)

Perform a Shapiro-Wilk test of the null hypothesis that the data in vector X come from a normal distribution.

This implementation is based on the method by Royston (1992). The calculation of the p-value is exact for sample size N = 3, and for ranges 4 ≤ N ≤ 11 and 12 ≤ N ≤ 5000 (Royston 1992) two separate approximations for p-values are used.

Keyword arguments

The following keyword arguments may be passed.

• sorted::Bool=issorted(X): to indicate that sample X is already sorted.
• censored::Integer=0: to censor the largest samples from X (so called upper-tail censoring)

Implements: pvalue

Implementation notes

• The current implementation DOES NOT implement p-values for censored data.
• If multiple Shapiro-Wilk tests are to be performed on samples of same size, it is faster to construct swc = shapiro_wilk_coefs(length(X)) once and pass it to the test via ShapiroWilkTest(X, swc) for re-use.
• For maximal performance sorted X should be passed and indicated with sorted=true keyword argument.

References

• Shapiro and Wilk (1965). An Analysis of Variance Test for Normality (Complete Samples)

• Royston (1992). Approximating the Shapiro-Wilk W-test for Non-Normality

• Royston (1993). A Toolkit for Testing for Non-Normality in Complete and Censored Samples

• Royston (1995). Remark AS R94: A Remark on Algorithm AS 181: The W-test for Normality