Statistical Modelling and Variable Selection in Climate Science
371
6 LASSO Regression Modelling
Lasso regression is more helpful in selecting a subset of “important” explanatory
variables from a pool of all the explanatory variables under consideration. This is
also referred to as “subset selection”. The ridge regression essentially re-scales the
ordinary least squares estimates. The LASSO, in contrast, tries to produce a sparse
solution, in the sense that several of the regression coefficients will be set to zero.
The meaning of sparse in this context is that when most of the elements are zero,
then it is termed as sparse. Ridge regression also tries to find the variables, whose
regression coefficients are nearly zero. One may therefore refer to ridge regression
as soft threshold, whereas the subset selection is a hard threshold; since, in the latter,
only a subset of the explanatory variables are included in the final model. The LASSO
minimization problem can be formulated as
ˆ
β ridge = arg min
β∈R k
n
i=1
(Y i − X
T
i β)
2
(29)
subject to
k
j=1 |β j | ≤ t
∗ for t
∗
≥ 0. The interpretation of this constraint is similar
to the constraint in ridge regression with a difference that now the sum of absolute
values of regression coefficients is considered whereas earlier the sum of squared
values of regression coefficient was considered.
This can again be re-formulated using the Lagrangian function for the L 1 -penalty
as follows:
ˆ
β lasso = arg min
β∈R k
⎧
⎨
⎩
n
i=1
(Y i − X
T
i β)
2
+ λ
k
j=1
|β j |
⎫
⎬
⎭
,
where λ ≥ 0, and as before, there exists a one-to-one correspondence between t
∗
and λ. Note that the subscript “1” in L 1 refers to the power of β j , which is here 1.
Also, in case of ridge regression, a closed-form solution was obtained, see, e.g., (27)
or (28). Contrary to the ridge regression, the LASSO does not admit a closed-form
solution. The L 1 -penalty makes the solution of (29) non-linear in the y i ’s. This type
of constrained minimization is a quadratic programming problem, whose solution
can be efficiently approximated.
6.1 Selecting the λ Value for Ridge and LASSO
The process of choosing λ ≥ 0 primarily depends upon the constraint
k
j=1 |β j | ≤
t
∗ . The choice of t
∗ plays a crucial role in selecting the subset of explanatory variables.
Larger value of t
∗ will select those variables, which are away from zero. Too small
values of λ can lead to overfitting when the model would tend to describe the random
371
6 LASSO Regression Modelling
Lasso regression is more helpful in selecting a subset of “important” explanatory
variables from a pool of all the explanatory variables under consideration. This is
also referred to as “subset selection”. The ridge regression essentially re-scales the
ordinary least squares estimates. The LASSO, in contrast, tries to produce a sparse
solution, in the sense that several of the regression coefficients will be set to zero.
The meaning of sparse in this context is that when most of the elements are zero,
then it is termed as sparse. Ridge regression also tries to find the variables, whose
regression coefficients are nearly zero. One may therefore refer to ridge regression
as soft threshold, whereas the subset selection is a hard threshold; since, in the latter,
only a subset of the explanatory variables are included in the final model. The LASSO
minimization problem can be formulated as
ˆ
β ridge = arg min
β∈R k
n
i=1
(Y i − X
T
i β)
2
(29)
subject to
k
j=1 |β j | ≤ t
∗ for t
∗
≥ 0. The interpretation of this constraint is similar
to the constraint in ridge regression with a difference that now the sum of absolute
values of regression coefficients is considered whereas earlier the sum of squared
values of regression coefficient was considered.
This can again be re-formulated using the Lagrangian function for the L 1 -penalty
as follows:
ˆ
β lasso = arg min
β∈R k
⎧
⎨
⎩
n
i=1
(Y i − X
T
i β)
2
+ λ
k
j=1
|β j |
⎫
⎬
⎭
,
where λ ≥ 0, and as before, there exists a one-to-one correspondence between t
∗
and λ. Note that the subscript “1” in L 1 refers to the power of β j , which is here 1.
Also, in case of ridge regression, a closed-form solution was obtained, see, e.g., (27)
or (28). Contrary to the ridge regression, the LASSO does not admit a closed-form
solution. The L 1 -penalty makes the solution of (29) non-linear in the y i ’s. This type
of constrained minimization is a quadratic programming problem, whose solution
can be efficiently approximated.
6.1 Selecting the λ Value for Ridge and LASSO
The process of choosing λ ≥ 0 primarily depends upon the constraint
k
j=1 |β j | ≤
t
∗ . The choice of t
∗ plays a crucial role in selecting the subset of explanatory variables.
Larger value of t
∗ will select those variables, which are away from zero. Too small
values of λ can lead to overfitting when the model would tend to describe the random