Multivariate Distributions

# Multivariate Distributions

Multivariate distributions are the distributions whose variate forms are Multivariate (i.e each sample is a vector). Abstract types for multivariate distributions:

const MultivariateDistribution{S<:ValueSupport} = Distribution{Multivariate,S}

const DiscreteMultivariateDistribution   = Distribution{Multivariate, Discrete}
const ContinuousMultivariateDistribution = Distribution{Multivariate, Continuous}

## Common Interface

The methods listed as below are implemented for each multivariate distribution, which provides a consistent interface to work with multivariate distributions.

### Computation of statistics

length(d::MultivariateDistribution) -> Int

Return the sample dimension of distribution d.

source
size(d::MultivariateDistribution)

Return the sample size of distribution d, i.e (length(d),).

source
eltype(d::MultivariateDistribution)

Return the sample type of distribution d

source
mean(d::MultivariateDistribution)

Compute the mean vector of distribution d.

source
var(d::MultivariateDistribution)

Compute the vector of element-wise variances for distribution d.

source
cov(d::MultivariateDistribution)

Compute the covariance matrix for distribution d. (cor is provided based on cov).

source
cor(d::MultivariateDistribution)

Computes the correlation matrix for distribution d.

source
entropy(d::MultivariateDistribution)

Compute the entropy value of distribution d.

source
entropy(d::MultivariateDistribution, b::Real)

Compute the entropy value of distribution $d$, w.r.t. a given base.

source

### Probability evaluation

insupport(d::MultivariateDistribution, x::AbstractArray)

If $x$ is a vector, it returns whether x is within the support of $d$. If $x$ is a matrix, it returns whether every column in $x$ is within the support of $d$.

source
pdf(d::MultivariateDistribution, x::AbstractArray)

Return the probability density of distribution d evaluated at x.

• If x is a vector, it returns the result as a scalar.
• If x is a matrix with n columns, it returns a vector r of length n, where r[i] corresponds

to x[:,i] (i.e. treating each column as a sample).

pdf!(r, d, x) will write the results to a pre-allocated array r.

source
logpdf(d::MultivariateDistribution, x::AbstractArray)

Return the logarithm of probability density evaluated at x.

• If x is a vector, it returns the result as a scalar.
• If x is a matrix with n columns, it returns a vector r of length n, where r[i] corresponds to x[:,i].

logpdf!(r, d, x) will write the results to a pre-allocated array r.

source
loglikelihood(d::MultivariateDistribution, x::AbstractMatrix)

The log-likelihood of distribution d w.r.t. all columns contained in matrix x.

source

Note: For multivariate distributions, the pdf value is usually very small or large, and therefore direct evaluating the pdf may cause numerical problems. It is generally advisable to perform probability computation in log-scale.

### Sampling

rand(::AbstractRNG, ::Sampleable)

Samples from the sampler and returns the result.

source
rand!([rng::AbstractRNG,] d::MultivariateDistribution, x::AbstractArray)

Draw samples and output them to a pre-allocated array x. Here, x can be either a vector of length dim(d) or a matrix with dim(d) rows.

source

Note: In addition to these common methods, each multivariate distribution has its own special methods, as introduced below.

## Distributions

The Multinomial distribution generalizes the binomial distribution. Consider n independent draws from a Categorical distribution over a finite set of size k, and let $X = (X_1, ..., X_k)$ where $X_i$ represents the number of times the element $i$ occurs, then the distribution of $X$ is a multinomial distribution. Each sample of a multinomial distribution is a k-dimensional integer vector that sums to n.

The probability mass function is given by

$f(x; n, p) = \frac{n!}{x_1! \cdots x_k!} \prod_{i=1}^k p_i^{x_i}, \quad x_1 + \cdots + x_k = n$
Multinomial(n, p)   # Multinomial distribution for n trials with probability vector p
Multinomial(n, k)   # Multinomial distribution for n trials with equal probabilities
# over 1:k
source

The Multivariate normal distribution is a multidimensional generalization of the normal distribution. The probability density function of a d-dimensional multivariate normal distribution with mean vector $\boldsymbol{\mu}$ and covariance matrix $\boldsymbol{\Sigma}$ is:

$f(\mathbf{x}; \boldsymbol{\mu}, \boldsymbol{\Sigma}) = \frac{1}{(2 \pi)^{d/2} |\boldsymbol{\Sigma}|^{1/2}} \exp \left( - \frac{1}{2} (\mathbf{x} - \boldsymbol{\mu})^T \Sigma^{-1} (\mathbf{x} - \boldsymbol{\mu}) \right)$

We realize that the mean vector and the covariance often have special forms in practice, which can be exploited to simplify the computation. For example, the mean vector is sometimes just a zero vector, while the covariance matrix can be a diagonal matrix or even in the form of $\sigma \mathbf{I}$. To take advantage of such special cases, we introduce a parametric type MvNormal, defined as below, which allows users to specify the special structure of the mean and covariance.

struct MvNormal{Cov<:AbstractPDMat,Mean<:AbstractVector} <: AbstractMvNormal
μ::Mean
Σ::Cov
end

Here, the mean vector can be an instance of any AbstractVector, including ZeroVector. The latter is simply an empty type indicating a vector filled with zeros. The covariance can be of any subtype of AbstractPDMat. Particularly, one can use PDMat for full covariance, PDiagMat for diagonal covariance, and ScalMat for the isotropic covariance – those in the form of $\sigma \mathbf{I}$. (See the Julia package PDMats for details).

We also define a set of alias for the types using different combinations of mean vectors and covariance:

const IsoNormal  = MvNormal{ScalMat,  Vector{Float64}}
const DiagNormal = MvNormal{PDiagMat, Vector{Float64}}
const FullNormal = MvNormal{PDMat,    Vector{Float64}}

const ZeroMeanIsoNormal  = MvNormal{ScalMat,  ZeroVector{Float64}}
const ZeroMeanDiagNormal = MvNormal{PDiagMat, ZeroVector{Float64}}
const ZeroMeanFullNormal = MvNormal{PDMat,    ZeroVector{Float64}}
source
MvNormal

Generally, users don't have to worry about these internal details. We provide a common constructor MvNormal, which will construct a distribution of appropriate type depending on the input arguments.

MvNormal(sig)

Construct a multivariate normal distribution with zero mean and covariance represented by sig.

MvNormal(mu, sig)

Construct a multivariate normal distribution with mean mu and covariance represented by sig.

MvNormal(d, sig)

Construct a multivariate normal distribution of dimension d, with zero mean, and an isotropic covariance matrix corresponding abs2(sig)*I.

Arguments

• mu::Vector{T<:Real}: The mean vector.
• d::Real: dimension of distribution.
• sig: The covariance, which can in of either of the following forms (with T<:Real):
1. subtype of AbstractPDMat,
2. symmetric matrix of type Matrix{T},
3. vector of type Vector{T}: indicating a diagonal covariance as diagm(abs2(sig)),
4. real-valued number: indicating an isotropic covariance matrix corresponding abs2(sig) * I.

Note: The constructor will choose an appropriate covariance form internally, so that special structure of the covariance can be exploited.

source
MvNormalCanon

Multivariate normal distribution is an exponential family distribution, with two canonical parameters: the potential vector $\mathbf{h}$ and the precision matrix $\mathbf{J}$. The relation between these parameters and the conventional representation (i.e. the one using mean $\boldsymbol{\mu}$ and covariance $\boldsymbol{\Sigma}$) is:

$\mathbf{h} = \boldsymbol{\Sigma}^{-1} \boldsymbol{\mu}, \quad \text{ and } \quad \mathbf{J} = \boldsymbol{\Sigma}^{-1}$

The canonical parameterization is widely used in Bayesian analysis. We provide a type MvNormalCanon, which is also a subtype of AbstractMvNormal to represent a multivariate normal distribution using canonical parameters. Particularly, MvNormalCanon is defined as:

struct MvNormalCanon{P<:AbstractPDMat,V<:Union{Vector,ZeroVector}} <: AbstractMvNormal
μ::V    # the mean vector
h::V    # potential vector, i.e. inv(Σ) * μ
J::P    # precision matrix, i.e. inv(Σ)
end

We also define aliases for common specializations of this parametric type:

const FullNormalCanon = MvNormalCanon{PDMat,    Vector{Float64}}
const DiagNormalCanon = MvNormalCanon{PDiagMat, Vector{Float64}}
const IsoNormalCanon  = MvNormalCanon{ScalMat,  Vector{Float64}}

const ZeroMeanFullNormalCanon = MvNormalCanon{PDMat,    ZeroVector{Float64}}
const ZeroMeanDiagNormalCanon = MvNormalCanon{PDiagMat, ZeroVector{Float64}}
const ZeroMeanIsoNormalCanon  = MvNormalCanon{ScalMat,  ZeroVector{Float64}}

A multivariate distribution with canonical parameterization can be constructed using a common constructor MvNormalCanon as:

MvNormalCanon(h, J)

Construct a multivariate normal distribution with potential vector h and precision matrix represented by J.

MvNormalCanon(J)

Construct a multivariate normal distribution with zero mean (thus zero potential vector) and precision matrix represented by J.

MvNormalCanon(d, J)

Construct a multivariate normal distribution of dimension d, with zero mean and an isotropic precision matrix corresponding J*I.

Arguments

• d::Int: dimension of distribution
• h::Vector{T<:Real}: the potential vector, of type Vector{T} with T<:Real.
• J: the representation of the precision matrix, which can be in either of the following forms (T<:Real):
1. an instance of a subtype of AbstractPDMat,
2. a square matrix of type Matrix{T},
3. a vector of type Vector{T}: indicating a diagonal precision matrix as diagm(J),
4. a real number: indicating an isotropic precision matrix corresponding J*I.

Note: MvNormalCanon share the same set of methods as MvNormal.

source
MvLogNormal(d::MvNormal)

The Multivariate lognormal distribution is a multidimensional generalization of the lognormal distribution.

If $\boldsymbol X \sim \mathcal{N}(\boldsymbol\mu,\,\boldsymbol\Sigma)$ has a multivariate normal distribution then $\boldsymbol Y=\exp(\boldsymbol X)$ has a multivariate lognormal distribution.

Mean vector $\boldsymbol{\mu}$ and covariance matrix $\boldsymbol{\Sigma}$ of the underlying normal distribution are known as the location and scale parameters of the corresponding lognormal distribution.

source
Dirichlet

The Dirichlet distribution is often used as the conjugate prior for Categorical or Multinomial distributions. The probability density function of a Dirichlet distribution with parameter $\alpha = (\alpha_1, \ldots, \alpha_k)$ is:

$f(x; \alpha) = \frac{1}{B(\alpha)} \prod_{i=1}^k x_i^{\alpha_i - 1}, \quad \text{ with } B(\alpha) = \frac{\prod_{i=1}^k \Gamma(\alpha_i)}{\Gamma \left( \sum_{i=1}^k \alpha_i \right)}, \quad x_1 + \cdots + x_k = 1$
# Let alpha be a vector
Dirichlet(alpha)         # Dirichlet distribution with parameter vector alpha

# Let a be a positive scalar
Dirichlet(k, a)          # Dirichlet distribution with parameter a * ones(k)
source
Product <: MultivariateDistribution

An N dimensional MultivariateDistribution constructed from a vector of N independent UnivariateDistributions.

Product(Uniform.(rand(10), 1)) # A 10-dimensional Product from 10 independent Uniform distributions.
source

### AbstractMvNormal

In addition to the methods listed in the common interface above, we also provide the following methods for all multivariate distributions under the base type AbstractMvNormal:

invcov(d::AbstractMvNormal)

Return the inversed covariance matrix of d.

source
logdetcov(d::AbstractMvNormal)

Return the log-determinant value of the covariance matrix.

source
sqmahal(d, x)

Return the squared Mahalanobis distance from x to the center of d, w.r.t. the covariance. When x is a vector, it returns a scalar value. When x is a matrix, it returns a vector of length size(x,2).

sqmahal!(r, d, x) with write the results to a pre-allocated array r.

source
rand(::AbstractRNG, ::Distributions.AbstractMvNormal)

Sample a random vector from the provided multi-variate normal distribution.

source

### MvLogNormal

In addition to the methods listed in the common interface above, we also provide the following methods:

location(d::MvLogNormal)

Return the location vector of the distribution (the mean of the underlying normal distribution).

source
scale(d::MvLogNormal)

Return the scale matrix of the distribution (the covariance matrix of the underlying normal distribution).

source
median(d::MvLogNormal)

Return the median vector of the lognormal distribution. which is strictly smaller than the mean.

source
mode(d::MvLogNormal)

Return the mode vector of the lognormal distribution, which is strictly smaller than the mean and median.

source

It can be necessary to calculate the parameters of the lognormal (location vector and scale matrix) from a given covariance and mean, median or mode. To that end, the following functions are provided.

location{D<:AbstractMvLogNormal}(::Type{D},s::Symbol,m::AbstractVector,S::AbstractMatrix)

Calculate the location vector (the mean of the underlying normal distribution).

• If s == :meancov, then m is taken as the mean, and S the covariance matrix of a lognormal distribution.
• If s == :mean | :median | :mode, then m is taken as the mean, median or mode of the lognormal respectively, and S is interpreted as the scale matrix (the covariance of the underlying normal distribution).

It is not possible to analytically calculate the location vector from e.g., median + covariance, or from mode + covariance.

source
location!{D<:AbstractMvLogNormal}(::Type{D},s::Symbol,m::AbstractVector,S::AbstractMatrix,μ::AbstractVector)

Calculate the location vector (as above) and store the result in $μ$

source
scale{D<:AbstractMvLogNormal}(::Type{D},s::Symbol,m::AbstractVector,S::AbstractMatrix)

Calculate the scale parameter, as defined for the location parameter above.

source
scale!{D<:AbstractMvLogNormal}(::Type{D},s::Symbol,m::AbstractVector,S::AbstractMatrix,Σ::AbstractMatrix)

Calculate the scale parameter, as defined for the location parameter above and store the result in Σ.

source
params{D<:AbstractMvLogNormal}(::Type{D},m::AbstractVector,S::AbstractMatrix)

Return (scale,location) for a given mean and covariance

source

## Internal Methods (for creating you own multivariate distribution)

_logpdf{T<:Real}(d::MultivariateDistribution, x::AbstractArray)

Evaluate logarithm of pdf value for a given vector x. This function need not perform dimension checking. Generally, one does not need to implement pdf (or _pdf) as fallback methods are provided in src/multivariates.jl.

source

## Product distributions

product_distribution(dists::AbstractVector{<:UnivariateDistribution})

Creates a multivariate product distribution P from a vector of univariate distributions. Fallback is the Product constructor, but specialized methods can be defined for distributions with a special multivariate product.

source
product_distribution(dists::AbstractVector{<:Normal})

Computes the multivariate Normal distribution obtained by stacking the univariate normal distributions. The result is a multivariate Gaussian with a diagonal covariance matrix.

source

Using product_distribution is advised to construct product distributions. For some distributions, it constructs a special multivariate type.