302
F. Firouzi et al.
X 1
Sum
X 2
X n
b
W
1
W
2
W
n
Error
Output
Sigmoid
Quantizer
Weights
Inputs
…
…
Fig. 5.55 Schematic presentation of a logistic regression model (a neural network with one
neuron)
Leaky ReLU Leaky ReLU is a workaround for the “dying ReLU” problem. A
leaky ReLU function is defined similar to a normal ReLU, but instead of the output
of zero in the negative region of the x-axis, it has a slight slope. In other words,
leaky ReLU provides a small, positive gradient when the input is below zero.
5.6.4 Softmax Function
Note that logistic regression can be seen as a simple neural network which has just
one neuron with sigmoid function as its activation function (see Fig. 5.55). Also
recall that logistic regression is indeed a two-class classifier which produces a value
between 0 and 1.0. Consider an email classifier as an example. When the output
of the logistic regression is 0.8, it suggests that the target email is spam with a
chance of 80%, and with a chance of 20% is not spam. Softmax regression through
a neural network layer extends this idea into a multiclass world. Softmax regression
is a general form of the logistic regression (Fig. 5.55), which enables us to perform
multiclass classification. This can improve the capability of conventional logistic
regression because logistic regression is only suitable for binary classification tasks.
In softmax regression, we have a neural network with several outputs in a way
that each output corresponds to one class. In softmax regression, we also need
to replace the sigmoid function of the output layer with softmax function (see
Fig. 5.56):
P
y = j | z
(i)
= φ sof tmax
z
(i)
=
e z (i)
k
j =0 e
z
(i)
k
F. Firouzi et al.
X 1
Sum
X 2
X n
b
W
1
W
2
W
n
Error
Output
Sigmoid
Quantizer
Weights
Inputs
…
…
Fig. 5.55 Schematic presentation of a logistic regression model (a neural network with one
neuron)
Leaky ReLU Leaky ReLU is a workaround for the “dying ReLU” problem. A
leaky ReLU function is defined similar to a normal ReLU, but instead of the output
of zero in the negative region of the x-axis, it has a slight slope. In other words,
leaky ReLU provides a small, positive gradient when the input is below zero.
5.6.4 Softmax Function
Note that logistic regression can be seen as a simple neural network which has just
one neuron with sigmoid function as its activation function (see Fig. 5.55). Also
recall that logistic regression is indeed a two-class classifier which produces a value
between 0 and 1.0. Consider an email classifier as an example. When the output
of the logistic regression is 0.8, it suggests that the target email is spam with a
chance of 80%, and with a chance of 20% is not spam. Softmax regression through
a neural network layer extends this idea into a multiclass world. Softmax regression
is a general form of the logistic regression (Fig. 5.55), which enables us to perform
multiclass classification. This can improve the capability of conventional logistic
regression because logistic regression is only suitable for binary classification tasks.
In softmax regression, we have a neural network with several outputs in a way
that each output corresponds to one class. In softmax regression, we also need
to replace the sigmoid function of the output layer with softmax function (see
Fig. 5.56):
P
y = j | z
(i)
= φ sof tmax
z
(i)
=
e z (i)
k
j =0 e
z
(i)
k