264
F. Firouzi et al.
Fig. 5.18 Graphical
presentation of the way that a
gradient algorithm works
J(W)
Initial
weights
W
Global minimum
4. Repeat steps 2–3 until some termination criteria (such as the cost J(W)) stop
reducing.
The update is performed for all of the training data, and the value of the gradient
depends on the current values of the model parameters and the cost function.
Parameter α is the learning rate, which controls the step size of each iteration.
This value α should be selected carefully because a high learning rate leads to
overshooting the minimum, and low rate results in reaching the minimum very
slowly. A good approach for a proper selection of learning rate is starting with small
values (such as 0.01 or 0.001) and redefining it based on the behavior of the gradient
descent algorithm.
Figure 5.18 presents an intuition of the gradient descent algorithm. In this
example, a blind man intended to reach the lowest altitude of rough terrain. One
of the simplest approaches for him is to feel the slope of the ground and move
in the direction that descends faster. If he keeps repeating this procedure, he will
reach the lowest altitude point of the terrain. In comparison to the gradient descent
algorithm, the slope is analogous to calculating the gradient; each step is similar to
each iteration, and the cost function is to find lower altitudes.
Gradient descent algorithm can also be formulated stochastically. Stochastic
gradient descent algorithm (SGD) computes the gradient by using a randomly
chosen training data point in each update (instead of considering all data points
of the training dataset). As a result, the algorithm runs faster, and yet, it moves in
the same direction over many updates as traditional gradient descent.
5.2.2 Regularization in Linear Regression
One of the major challenges in regression analysis is overfitting/underfitting.
Overfitting (high variance) occurs when the regression analysis algorithm works
with a high level of efficiency on the training dataset but fails to perform correct
F. Firouzi et al.
Fig. 5.18 Graphical
presentation of the way that a
gradient algorithm works
J(W)
Initial
weights
W
Global minimum
4. Repeat steps 2–3 until some termination criteria (such as the cost J(W)) stop
reducing.
The update is performed for all of the training data, and the value of the gradient
depends on the current values of the model parameters and the cost function.
Parameter α is the learning rate, which controls the step size of each iteration.
This value α should be selected carefully because a high learning rate leads to
overshooting the minimum, and low rate results in reaching the minimum very
slowly. A good approach for a proper selection of learning rate is starting with small
values (such as 0.01 or 0.001) and redefining it based on the behavior of the gradient
descent algorithm.
Figure 5.18 presents an intuition of the gradient descent algorithm. In this
example, a blind man intended to reach the lowest altitude of rough terrain. One
of the simplest approaches for him is to feel the slope of the ground and move
in the direction that descends faster. If he keeps repeating this procedure, he will
reach the lowest altitude point of the terrain. In comparison to the gradient descent
algorithm, the slope is analogous to calculating the gradient; each step is similar to
each iteration, and the cost function is to find lower altitudes.
Gradient descent algorithm can also be formulated stochastically. Stochastic
gradient descent algorithm (SGD) computes the gradient by using a randomly
chosen training data point in each update (instead of considering all data points
of the training dataset). As a result, the algorithm runs faster, and yet, it moves in
the same direction over many updates as traditional gradient descent.
5.2.2 Regularization in Linear Regression
One of the major challenges in regression analysis is overfitting/underfitting.
Overfitting (high variance) occurs when the regression analysis algorithm works
with a high level of efficiency on the training dataset but fails to perform correct