Nonlinear dimensionality reduction
|
|
High dimensional data can be difficult to interpret. One approach is to assume that the data lies on an embedded non-linear manifold. If that manifold is of low enough dimension then the data can be visualised in the low dimensional space.
Below we summarise some important algorithms in the history of manifold learning and nonlinear dimensionality reduction. Many of these non-linear dimensionality reduction methods are related to linear methods which we list below. The non-linear methods can be broadly classified into two groups. Those which actually provide a mapping (either from the high dimensional space to the low dimensional embedding or vice versa) and those that just give a visualisation. Typically those that just give a visualisation are based on proximity data.
| Contents |
Linear methods
- Independent component analysis (ICA).
- Principal component analysis (PCA) (also called Karhunen-Loève transform — KLT).
- Singular value decomposition (SVD).
- Factor analysis.
Non-linear mappings
Perhaps the principal method amongst those that provide a mapping from the high dimensional space to the embedded space is kernel PCA. This method provides a non-linear PCAthrought the use of kernel functions.
NeuroScale uses stress functions inspired by multidimensional scaling and Sammon mappings (see below) to learn a non-linear mapping from the high dimensional to the embedded space. The mappings in NeuroScale are based on neural network models.
The generative topographic mapping (GTM) uses a point representation in the embedded space to form a latent variable model which is based around a non-linear mapping from the embedded space to the high dimensional space. It was inspired by work on density networks which also are based around the same probabilistic model.
Gaussian process latent variable models (GPLVM) are a probabilistic non-linear PCA. Like kernel PCA they use a kernel function to form the mapping (in the form of a Gaussian process). However in the GPLVM the mapping is from the embedded space to the data space (like density networks and GTM) whereas in kernel PCA it is in the opposite direction.
Other methods include
- Neural network methods.
- Nonlinear autoencoders.
Methods based on proximity matrices
A method based on proximity matrices is one where the data is presented to the algorithm in the form of a similarity matrix or a distance matrix. These methods all fall under the broader class of multidimensional scaling. The variations tend to be differences in how the proximity data is computed, for example Isomap, locally linear embeddings are examples of metric multidimensional scaling and the Sammon mapping (which is not in fact a mapping) is an example of a non-metric multidimensional scaling method.
See also
External links
- Isomap (http://isomap.stanford.edu/)
- Generative Topographic Mapping (http://www.ncrg.aston.ac.uk/GTM/)
- Mike Tipping's Thesis (http://research.microsoft.com/users/mtipping/pages/thesis.htm)
- Gaussian Process Latent Variable Model (http://www.dcs.shef.ac.uk/~neil/gplvm/)
- Locally Linear Embedding (http://www.cs.toronto.edu/~roweis/lle/)
- Semidefinite Embedding (http://www.seas.upenn.edu/~kilianw/sde)