Gaussian Process - Gaussian Process For Machine Learning

In probability theory and statistics, a Gaussian process is a particular kind of statistical model where observations occur in a continuous domain, e.g. time or space. In a Gaussian process, every point in some continuous input space is associated with a normally distributed random variable. Moreover, every finite collection of those random variables has a multivariate normal distribution, i.e. every finite linear combination of them is normally distributed. The distribution of a Gaussian process is the joint distribution of all those (infinitely many) random variables, and as such, it is a distribution over functions with a continuous domain, e.g. time or space.

Viewed as a machine-learning algorithm, a Gaussian process uses lazy learning and a measure of the similarity between points (the kernel function) to predict the value for an unseen point from training data. The prediction is not just an estimate for that point, but also has uncertainty informationâ€"it is a one-dimensional Gaussian distribution (which is the marginal distribution at that point).

For some kernel functions, matrix algebra can be used to calculate the predictions using the technique of kriging. When a parameterised kernel is used, optimisation software is typically used to fit a Gaussian process model.

The concept of Gaussian processes is named after Carl Friedrich Gauss because it is based on the notion of the Gaussian distribution (normal distribution). Gaussian processes can be seen as an infinite-dimensional generalization of multivariate normal distributions.

Gaussian processes are useful in statistical modelling, benefiting from properties inherited from the normal. For example, if a random process is modelled as a Gaussian process, the distributions of various derived quantities can be obtained explicitly. Such quantities include the average value of the process over a range of times and the error in estimating the average using sample values at a small set of times.

Definition

source : www.cs.toronto.edu

A time continuous stochastic process is Gaussian if and only if for every finite set of indices ${\displaystyle t_{1},\ldots ,t_{k}}$ in the index set ${\displaystyle T}$

${\displaystyle \mathbf {X} _{t_{1},\ldots ,t_{k}}=(\mathbf {X} _{t_{1}},\ldots ,\mathbf {X} _{t_{k}})}$

is a multivariate Gaussian random variable. That is the same as saying every linear combination of ${\displaystyle (\mathbf {X} _{t_{1}},\ldots ,\mathbf {X} _{t_{k}})}$ has a univariate normal (or Gaussian) distribution. Using characteristic functions of random variables, the Gaussian property can be formulated as follows: ${\displaystyle \left\{X_{t};t\in T\right\}}$ is Gaussian if and only if, for every finite set of indices ${\displaystyle t_{1},\ldots ,t_{k}}$, there are real-valued ${\displaystyle \sigma _{\ell j}}$, ${\displaystyle \mu _{\ell }}$ with ${\displaystyle \sigma _{jj}>0}$ such that the following equality holds for all ${\displaystyle s_{1},s_{2},\ldots ,s_{k}\in \mathbb {R} }$

${\displaystyle \operatorname {E} \left(\exp \left(i\ \sum _{\ell =1}^{k}s_{\ell }\ \mathbf {X} _{t_{\ell }}\right)\right)=\exp \left(-{\frac {1}{2}}\,\sum _{\ell ,j}\sigma _{\ell j}s_{\ell }s_{j}+i\sum _{\ell }\mu _{\ell }s_{\ell }\right).}$

where ${\displaystyle i}$ denotes the imaginary number ${\displaystyle {\sqrt {-1}}}$.

The numbers ${\displaystyle \sigma _{\ell j}}$ and ${\displaystyle \mu _{\ell }}$ can be shown to be the covariances and means of the variables in the process.

Covariance functions

source : www.robots.ox.ac.uk

A key fact of Gaussian processes is that they can be completely defined by their second-order statistics. Thus, if a Gaussian process is assumed to have mean zero, defining the covariance function completely defines the process' behaviour. Importantly the non-negative definiteness of this function enables its spectral decomposition using the Karhunenâ€"Loeve expansion. Basic aspects that can be defined through the covariance function are the process' stationarity, isotropy, smoothness and periodicity.

Stationarity refers to the process' behaviour regarding the separation of any two points x and x' . If the process is stationary, it depends on their separation, x âˆ' x', while if non-stationary it depends on the actual position of the points x and x'. For example, the special case of an Ornsteinâ€"Uhlenbeck process, a Brownian motion process, is stationary.

If the process depends only on |x âˆ' x'|, the Euclidean distance (not the direction) between x and x', then the process is considered isotropic. A process that is concurrently stationary and isotropic is considered to be homogeneous; in practice these properties reflect the differences (or rather the lack of them) in the behaviour of the process given the location of the observer.

Ultimately Gaussian processes translate as taking priors on functions and the smoothness of these priors can be induced by the covariance function. If we expect that for "near-by" input points x and x' their corresponding output points y and y' to be "near-by" also, then the assumption of continuity is present. If we wish to allow for significant displacement then we might choose a rougher covariance function. Extreme examples of the behaviour is the Ornsteinâ€"Uhlenbeck covariance function and the squared exponential where the former is never differentiable and the latter infinitely differentiable.

Periodicity refers to inducing periodic patterns within the behaviour of the process. Formally, this is achieved by mapping the input x to a two dimensional vector u(x) = (cos(x), sin(x)).

Usual covariance functions

There are a number of common covariance functions:

• Constant : ${\displaystyle K_{\operatorname {C} }(x,x')=C}$
• Linear: ${\displaystyle K_{\operatorname {L} }(x,x')=x^{T}x'}$
• Gaussian noise: ${\displaystyle K_{\operatorname {GN} }(x,x')=\sigma ^{2}\delta _{x,x'}}$
• Squared exponential: ${\displaystyle K_{\operatorname {SE} }(x,x')=\exp {\Big (}-{\frac {\|d\|^{2}}{2\ell ^{2}}}{\Big )}}$
• Ornsteinâ€"Uhlenbeck: ${\displaystyle K_{\operatorname {OU} }(x,x')=\exp \left(-{\frac {|d|}{\ell }}\right)}$
• Matérn: ${\displaystyle K_{\operatorname {Matern} }(x,x')={\frac {2^{1-\nu }}{\Gamma (\nu )}}{\Big (}{\frac {{\sqrt {2\nu }}|d|}{\ell }}{\Big )}^{\nu }K_{\nu }{\Big (}{\frac {{\sqrt {2\nu }}|d|}{\ell }}{\Big )}}$
• Periodic: ${\displaystyle K_{\operatorname {P} }(x,x')=\exp \left(-{\frac {2\sin ^{2}\left({\frac {d}{2}}\right)}{\ell ^{2}}}\right)}$
• Rational quadratic: ${\displaystyle K_{\operatorname {RQ} }(x,x')=(1+|d|^{2})^{-\alpha },\quad \alpha \geq 0}$

Here ${\displaystyle d=x-x'}$. The parameter â„" is the characteristic length-scale of the process (practically, "how close" two points ${\displaystyle x}$ and ${\displaystyle x'}$ have to be to influence each other significantly), δ is the Kronecker delta and σ the standard deviation of the noise fluctuations. Moreover, ${\displaystyle K_{\nu }}$ is the modified Bessel function of order ${\displaystyle \nu }$ and ${\displaystyle \Gamma (\nu )}$ is the gamma function evaluated at ${\displaystyle \nu }$. Importantly, a complicated covariance function can be defined as a linear combination of other simpler covariance functions in order to incorporate different insights about the data-set at hand.

Clearly, the inferential results are dependent on the values of the hyperparameters θ (e.g. â„" and σ) defining the model's behaviour. A popular choice for θ is to provide maximum a posteriori (MAP) estimates of it with some chosen prior. If the prior is very near uniform, this is the same as maximizing the marginal likelihood of the process; the marginalization being done over the observed process values ${\displaystyle y}$. This approach is also known as maximum likelihood II, evidence maximization, or empirical Bayes.

Brownian Motion as the Integral of Gaussian processes

source : link.springer.com

A Wiener process (aka Brownian motion) is the integral of a white noise Gaussian process. It is not stationary, but it has stationary increments.

The Ornsteinâ€"Uhlenbeck process is a stationary Gaussian process.

The Brownian bridge is (like the Ornsteinâ€"Uhlenbeck process) an example of a Gaussian process whose increments are not independent.

The fractional Brownian motion is a Gaussian process whose covariance function is a generalisation of that of the Wiener process.

Applications

source : slideplayer.com

A Gaussian process can be used as a prior probability distribution over functions in Bayesian inference. Given any set of N points in the desired domain of your functions, take a multivariate Gaussian whose covariance matrix parameter is the Gram matrix of your N points with some desired kernel, and sample from that Gaussian.

Inference of continuous values with a Gaussian process prior is known as Gaussian process regression, or kriging; extending Gaussian process regression to multiple target variables is known as cokriging. Gaussian processes are thus useful as a powerful non-linear multivariate interpolation tool. Gaussian process regression can be further extended to address learning tasks in both supervised (e.g. probabilistic classification) and unsupervised (e.g. manifold learning) learning frameworks.

Gaussian processes can also be used in the context of mixture of experts models, e.g.,. The underlying rationale of such a learning framework consists in the fundamental assumption that the mapping of independent to dependent variables cannot be sufficiently captured by a single Gaussian process model. On the contrary, it is considered that the observations space is naturally divided into subspaces, each of which is characterized by a significantly different mapping function; each of these is learned via a different Gaussian process component in the postulated mixture.

Gaussian process prediction, or kriging

When concerned with a general Gaussian process regression problem (kriging), it is assumed that for a Gaussian process f observed at coordinates x, the vector of values ${\displaystyle f(x)}$ is just one sample from a multivariate Gaussian distribution of dimension equal to number of observed coordinates |x|. Therefore, under the assumption of a zero-mean distribution, ${\displaystyle f(x)\sim N(0,K(\theta ,x,x'))}$, where ${\displaystyle K(\theta ,x,x')}$ is the covariance matrix between all possible pairs ${\displaystyle (x,x')}$ for a given set of hyperparameters θ. As such the log marginal likelihood is:

${\displaystyle \log p(f(x)|\theta ,x)=-{\frac {1}{2}}f(x)^{T}K(\theta ,x,x')^{-1}f(x)-{\frac {1}{2}}\log \det(K(\theta ,x,x'))-{\frac {|x|}{2}}\log 2\pi }$

and maximizing this marginal likelihood towards θ provides the complete specification of the Gaussian process f. One can briefly note at this point that the first term corresponds to a penalty term for a model's failure to fit observed values and the second term to a penalty term that increases proportionally to a model's complexity. Having specified θ making predictions about unobserved values ${\displaystyle f(x^{*})}$ at coordinates x* is then only a matter of drawing samples from the predictive distribution ${\displaystyle p(y^{*}\mid x^{*},f(x),x)=N(y^{*}\mid A,B)}$ where the posterior mean estimate A is defined as

${\displaystyle A=K(\theta ,x^{*},x)K(\theta ,x,x')^{-1}f(x)}$

and the posterior variance estimate B is defined as:

${\displaystyle B=K(\theta ,x^{*},x^{*})-K(\theta ,x^{*},x)K(\theta ,x,x')^{-1}K(\theta ,x^{*},x)^{T}}$

where ${\displaystyle K(\theta ,x^{*},x)}$ is the covariance between the new coordinate of estimation x* and all other observed coordinates x for a given hyperparameter vector θ, ${\displaystyle K(\theta ,x,x')}$ and ${\displaystyle f(x)}$ are defined as before and ${\displaystyle K(\theta ,x^{*},x^{*})}$ is the variance at point x* as dictated by θ. It is important to note that practically the posterior mean estimate ${\displaystyle f(x^{*})}$ (the "point estimate") is just a linear combination of the observations ${\displaystyle f(x)}$; in a similar manner the variance of ${\displaystyle f(x^{*})}$ is actually independent of the observations ${\displaystyle f(x)}$. A known bottleneck in Gaussian process prediction is that the computational complexity of prediction is cubic in the number of points |x| and as such can become unfeasible for larger data sets. Works on sparse Gaussian processes, that usually are based on the idea of building a representative set for the given process f, try to circumvent this issue.

See also

• Bayes linear statistics
• Bayesian interpretation of regularization
• Kriging
• Gradient-Enhanced Kriging (GEK)

Notes

External links

• www.GaussianProcess.com
• The Gaussian Processes Web Site, including the text of Rasmussen and Williams' Gaussian Processes for Machine Learning
• A gentle introduction to Gaussian processes
• A Review of Gaussian Random Fields and Correlation Functions

Software

• STK: a Small (Matlab/Octave) Toolbox for Kriging and GP modeling
• Kriging module in UQLab framework (Matlab)
• Matlab/Octave function for stationary Gaussian fields
• Yelp MOE â€" A black box optimization engine using Gaussian process learning
• ooDACE â€" A flexible object-oriented Kriging matlab toolbox.
• GPstuff â€" Gaussian process toolbox for Matlab and Octave
• GPy â€" A Gaussian processes framework in Python
• Interactive Gaussian process regression demo
• Basic Gaussian process library written in C++11
• scikit-learn â€" A machine learning library for Python which includes Gaussian process regression and classification

Video tutorials

• Gaussian Process Basics by David MacKay
• Learning with Gaussian Processes by Carl Edward Rasmussen
• Bayesian inference and Gaussian processes by Carl Edward Rasmussen