372
Shalabh and S. S. Dhar
errors or noise in the data. On the contrary, too large values of λ would lead to under
fitting when the procedure cannot capture the underlying relationship. In both cases,
we will get a high error value when calculated on the test data.
Cross-validation is one of the most powerful techniques that can be used to find
a “most suitable” value for the λ for a given data. By “most suitable” here we mean
that we are trying to find λ that would allow us to predict the values of study variable
with the highest accuracy.
To perform the cross-validation, the initial data is divided into two subsets: one is
called the training set and the other one is called the test set. The training set then is
used to calculate the coefficient estimates. These estimates are then validated on the
test set. Let us now describe the algorithm in some more detail. At first, the initial data
set is randomly divided into B blocks of equal length. One of the blocks is assigned
the role of the test set while the remaining (B − 1) blocks together constitute the
training set. In practice, the number of blocks B is usually selected to be 5 or 10.
Next, we choose a grid of values λ = λ s and calculate the regression coefficients
for each λ s value. Given these regression coefficients, we then compute the residual
sum of squares:
RSS λ s ,k =
n
i=1
⎛
⎝ y i −
k
j=1
ˆ
β(k, λ s )x i j
⎞
⎠ ,
(30)
where k = 1, 2, . . . , B is the index of the block selected as the test set. One can
obtain the average of these RSS values over all blocks as follows:
M SE λ s =
1
B
B
k=1
RSS λ s ,k .
(31)
Finally, λ is then set equal to λ s that gives the minimum M SE λ s .
6.2 Implementation of Ridge Regression and LASSO Using R
To compute the coefficient vectors β using the technique of ridge regression and
LASSO, one can use R software.
Example (contd.): We consider the earlier Example in Sect. 3, where the rainfall
during the monsoon season was assumed to depend on three variables viz., wind
speed, precipitation and relative humidity. Now three more variables, which we
expect that they affect the rainfall are added. The new variables are number of cars
per square kilometer in a city, population density (number of persons per square
kilometer) in a city and aluminum toxicity in soil (in ppm × 10
−2 ) and the full data
is given in Table 2. Note that the data on earlier variables, viz., y, x 1i , x 2i and x 3i
remains unchanged.
Shalabh and S. S. Dhar
errors or noise in the data. On the contrary, too large values of λ would lead to under
fitting when the procedure cannot capture the underlying relationship. In both cases,
we will get a high error value when calculated on the test data.
Cross-validation is one of the most powerful techniques that can be used to find
a “most suitable” value for the λ for a given data. By “most suitable” here we mean
that we are trying to find λ that would allow us to predict the values of study variable
with the highest accuracy.
To perform the cross-validation, the initial data is divided into two subsets: one is
called the training set and the other one is called the test set. The training set then is
used to calculate the coefficient estimates. These estimates are then validated on the
test set. Let us now describe the algorithm in some more detail. At first, the initial data
set is randomly divided into B blocks of equal length. One of the blocks is assigned
the role of the test set while the remaining (B − 1) blocks together constitute the
training set. In practice, the number of blocks B is usually selected to be 5 or 10.
Next, we choose a grid of values λ = λ s and calculate the regression coefficients
for each λ s value. Given these regression coefficients, we then compute the residual
sum of squares:
RSS λ s ,k =
n
i=1
⎛
⎝ y i −
k
j=1
ˆ
β(k, λ s )x i j
⎞
⎠ ,
(30)
where k = 1, 2, . . . , B is the index of the block selected as the test set. One can
obtain the average of these RSS values over all blocks as follows:
M SE λ s =
1
B
B
k=1
RSS λ s ,k .
(31)
Finally, λ is then set equal to λ s that gives the minimum M SE λ s .
6.2 Implementation of Ridge Regression and LASSO Using R
To compute the coefficient vectors β using the technique of ridge regression and
LASSO, one can use R software.
Example (contd.): We consider the earlier Example in Sect. 3, where the rainfall
during the monsoon season was assumed to depend on three variables viz., wind
speed, precipitation and relative humidity. Now three more variables, which we
expect that they affect the rainfall are added. The new variables are number of cars
per square kilometer in a city, population density (number of persons per square
kilometer) in a city and aluminum toxicity in soil (in ppm × 10
−2 ) and the full data
is given in Table 2. Note that the data on earlier variables, viz., y, x 1i , x 2i and x 3i
remains unchanged.