Statistical Modelling and Variable Selection in Climate Science
369
from zero. Needless to mention that the LASSO is computational and algorithmbased technique. Various versions, extensions, developments and improvements in
LASSO have been developed. In a general framework, such regression is called as
regularized regression. Now we give a brief application-oriented exposition of such
techniques. Our objective is to introduce LASSO but a brief introduction to ridge
regression is required to understand the philosophy of LASSO.
5 Ridge Regression Modelling
As mentioned in Sect. 2, we very often face a challenge of having large number
of explanatory variables in practice. However, many of them may not be impactful
on the response variable. To overcome this problem in practice, one may choose
only the significant explanatory variables and conduct the analysis based only on
those covariates. In literature, various regularized regression methodologies have
been proposed to solve this problem, and one such popular form of regularized
regression is ridge regression. Ridge regression is motivated by a limited problem of
minimization (constrained), which places a constraint on the sum of squares of the
coefficient’s weights and can be formulated as follows:
ˆ
β ridge = arg min
β∈R k
n
i=1
(Y i − X
T
i β)
2
(24)
subject to
k
j=1 β
2
j ≤ t
∗ for t
∗
≥ 0. The feasible set for this minimization
problem is therefore constrained to be S(t
∗
) = {β ∈ R
k
: ||β|| ≤ t
∗
}, where
β = (β 1 , β 2 , . . . , β k )
T does not include the intercept β 0 . Note that the L 2 -penalty
refers to the constraints
k
j=1 β
2
j ≤ t
∗ , which mean that the coefficients are not estimated freely but the estimated values have to satisfy the condition
k
j=1 β
2
j ≤ t
∗ ,
where t
∗ is a given value. Note that the subscript 2 in L 2 corresponds to superscript
2 (or square) in β
2
j . The ridge estimators are not equivariant under a re-scaling of the
x j ‘s, because of the L 2 -penalty. This issue is overcome by centering the explanatory
variables. Using a Lagrange multiplier technique for constrained optimization, this
problem of finding out the value of regression coefficients β under the constraints
k
j=1 β
2
j ≤ t
∗ can be alternatively formulated as
ˆ
β ridge = arg min
β∈R k
⎧
⎨
⎩
n
i=1
(Y i − X
T
i β)
2
+ λ
k
j=1
β
2
j
⎫
⎬
⎭
.
(25)
In fact, the residual sum of squares (i.e., like SS reg in multiple linear regression)
in ridge regression can be expressed as
369
from zero. Needless to mention that the LASSO is computational and algorithmbased technique. Various versions, extensions, developments and improvements in
LASSO have been developed. In a general framework, such regression is called as
regularized regression. Now we give a brief application-oriented exposition of such
techniques. Our objective is to introduce LASSO but a brief introduction to ridge
regression is required to understand the philosophy of LASSO.
5 Ridge Regression Modelling
As mentioned in Sect. 2, we very often face a challenge of having large number
of explanatory variables in practice. However, many of them may not be impactful
on the response variable. To overcome this problem in practice, one may choose
only the significant explanatory variables and conduct the analysis based only on
those covariates. In literature, various regularized regression methodologies have
been proposed to solve this problem, and one such popular form of regularized
regression is ridge regression. Ridge regression is motivated by a limited problem of
minimization (constrained), which places a constraint on the sum of squares of the
coefficient’s weights and can be formulated as follows:
ˆ
β ridge = arg min
β∈R k
n
i=1
(Y i − X
T
i β)
2
(24)
subject to
k
j=1 β
2
j ≤ t
∗ for t
∗
≥ 0. The feasible set for this minimization
problem is therefore constrained to be S(t
∗
) = {β ∈ R
k
: ||β|| ≤ t
∗
}, where
β = (β 1 , β 2 , . . . , β k )
T does not include the intercept β 0 . Note that the L 2 -penalty
refers to the constraints
k
j=1 β
2
j ≤ t
∗ , which mean that the coefficients are not estimated freely but the estimated values have to satisfy the condition
k
j=1 β
2
j ≤ t
∗ ,
where t
∗ is a given value. Note that the subscript 2 in L 2 corresponds to superscript
2 (or square) in β
2
j . The ridge estimators are not equivariant under a re-scaling of the
x j ‘s, because of the L 2 -penalty. This issue is overcome by centering the explanatory
variables. Using a Lagrange multiplier technique for constrained optimization, this
problem of finding out the value of regression coefficients β under the constraints
k
j=1 β
2
j ≤ t
∗ can be alternatively formulated as
ˆ
β ridge = arg min
β∈R k
⎧
⎨
⎩
n
i=1
(Y i − X
T
i β)
2
+ λ
k
j=1
β
2
j
⎫
⎬
⎭
.
(25)
In fact, the residual sum of squares (i.e., like SS reg in multiple linear regression)
in ridge regression can be expressed as