ppt - iitk.ac.in
Atul Singh Junior Undergraduate CSE, IIT Kanpur Dimension reduction is a technique which is used to represent a high dimensional data in a more compact way using some alternate representation Many process of data generation generate a large data set which is embedded in a low dimensional manifold of a high dimensional space. Dimensional reduction can be applied to those data sets In statistical pattern recognition, we often encounter a data set in a high dimensional space Many times the data is correlated in such a manner that there are very few independent dimensions Possible to represent the data using much lower dimensions. Some benefits are ◦ ◦ ◦ ◦ Compact representation Less processing time Visualization of high dimensional data Interpolate meaningful dimensions Linear Methods ◦ Principal Component Analysis (PCA) ◦ Independent Component Analysis (ICA) ◦ Multi-dimensional Scaling (MDS) Non-linear Methods ◦ Global Isomap and its variants ◦ Local Locally Linear Embedding (LLE) Laplacian Eigenmaps Principal Component Analysis ◦ Involves finding directions having large covariance ◦ Express the data as a linear combination of eigenvectors along those directions Multidimensional Scaling ◦ Keep inter point distances as invariant ◦ Again a linear methodology Many variables can’t be represented as linear combinations of some vectors ◦ Examples – Swiss roll, faces data, etc In general the low dimension data is embedded on some non linear manifold Not possible to transform these manifolds to low dimension space using only translation, rotation, rescaling Linear methods have a pre conceived notion of dimension reduction Goal is to – automate the estimation (infer the degrees of freedom of data manifold) and classification (embed data in low dimension space) process So need to go beyond Linear methods Non Linear Methods ◦ Isomap ◦ Locally Linear Embedding Global manifold learning algorithm A swiss roll (fig a) embedded as a manifold in high dimension space. Goal is to reduce it to two dimensions (fig c) Consider a human face ◦ How is it represented/stored in brain? ◦ How is it represented/stored in a computer? Do we need to store all the information (every pixel)? Just need to figure out some important structures Basic steps involved are ◦ Construct Neighborhood Graph: Determine the neighbors of each point and assign edge weights dx (i, j) to the graph thus formed ◦ Compute Geodesic distances: Estimate the geodesic distances dM (i, j) using Dijkstra’s algorithm ◦ Use MDS to lower dimensions: Apply MDS on the computed shortest path distance matrix and thus reduce the dimensionality So basically similar to MDS just using geodesic distances rather than Euclidean Another non-linear dimension reduction algorithm Follows a local approach to reduce the dimensions The idea is based on the assumption that a point on a manifold reside on a hyper plane determined by the point and its some nearest neighbors Key Question – How to combine these parts of hyper planes and map them to low dimensional space? The basic steps involved are ◦ Assign neighbors: To each point assign neighbors using nearest neighbor approach ◦ Weight calculation: We compute weights Wij such that Xi is best reconstructed from its neighbors ◦ Compute Low dimension embedding: Using the above computed weight matrix find corresponding embedding vectors Yi in lower dimensional space minimizing error function The weights computed for reconstruction are invariant with respect to translation, rotation and rescaling The same weights should reconstruct the map in reduced dimensions So we can conclude that the local geometry is preserved Shortcomings of Isomap ◦ Need a dense sampling of data on the manifold ◦ If k is chosen very small then residual error will be too large ◦ If k is chosen very large then short-circuiting may happen Shortcomings of LLE ◦ Due to local nature doesn’t give a complete picture of the data ◦ Again problems with selection of k Short-circuiting ◦ “When the distance between the folds is very less or there is noise such that a point from a different fold is chosen to be a neighbour of the point, the distance computed does not represent geodesic distance and hence the algorithm fails” Insight ◦ This problem arises due to selection of neighbors just on the basis of their Euclidean distance. The basic selection criteria are Select all the points within a ball of radius ε Select K nearest neighbors ◦ Locally Linear isomaps overcomes this problem by modifying the neighbor selection criteria Proposed algorithm – KLL Isomaps Similar to Tanenbaum’s Isomap except for the selection of nearest neighbors Previous algorithms (Isomap, LLE) consider only the Euclidean distance to judge neighborhood criteria The proposed algorithm is ◦ Find a candidate neighborhood using K-nn approach ◦ Construct the data point using candidate neighbors (as in LLE) to minimize the reconstruction error ◦ The neighbours whose Euclidean distance is less and those lying on the locally linear patch of the manifold get higher weights and hence are selected preferably ◦ Now KLL ≤ K neighbors are chosen based on the post construction weights KLL Isomap has been demonstrated to perform better than Isomap under ◦ Sparsely sampled data ◦ Noisy data ◦ Dense data without noise The metric used for Analysis are “Short circuit edges” and “Residual variance” To establish a formal proof for better performance of this algorithm Tenenbaum J.B., Silva V.d., Langford J.C.: A Global Geometric Framework for Nonlinear Dimensionality Reduction Roweis S.T., Saul L.K.: Nonlinear Dimensionality Reduction by Locally Linear Embedding Balasubramanian M., Schwartz E.L., Tenenbaum J.B., Silva V.d., Langford J.C.: The Isomap Algorithm and Topological Stability Silva V.d., Tenenbaum J.B.: Global versus local methods in nonlinear dimensionality reduction Roweis S.T., Saul L.K.: An Introduction to Locally Linear Embedding Saxena A., Gupta A., Mukherjee A.: Non-linear Dimensionality Reduction by Locally Linear Isomaps
© Copyright 2024