Seonglae ChoPartial Least Squares Regression
Performs "dimension reduction + regression simultaneously"
In other words, the key point is that B is not a simple least-squares solution, but rather a linear transformation based on covariance maximization.
A regression method that simultaneously optimizes the covariance structure between input features and output targets in high-dimensional data with severe Multicollinearity.
X is colinear and high-dimensional, so we cannot directly use as OLS. Therefore, we find the direction with the highest covariance, remove the explained part, and iteratively find the next direction in the residual.
Optimization
- The first latent direction explains part of : where is the remaining residual.
- The next component tries to explain the residual :
- This process repeats iteratively until becomes sufficiently small.
K
Determines how many latent axes to use
Partial least squares regression
Partial least squares (PLS) regression is a statistical method that bears some relation to principal components regression and is a reduced rank regression;[1] instead of finding hyperplanes of maximum variance between the response and independent variables, it finds a linear regression model by projecting the predicted variables and the observable variables to a new space of maximum covariance (see below). Because both the X and Y data are projected to new spaces, the PLS family of methods are known as bilinear factor models. Partial least squares discriminant analysis (PLS-DA) is a variant used when the Y is categorical.
https://en.wikipedia.org/wiki/Partial_least_squares_regression