234
Y. Zhou et al.
weighted average for the gradient, which is beneficial to correct the swing amplitude and speed up the convergence. The Adam algorithm combines the Momentum algorithm and the RMSProp algorithm, thus, it has the advantages of both
algorithms.
At the beginning of the training, gradient cumulates (v dω and v db ) and
squared cumulates (s dω and s db ) are initialized as:
v dω = v db = s dω = s db = 0.
(1)
Assume that in the tth training, the parameter updates for Momentum and
RMSProp are calculated by:
v dω = β 1 v dω + (1 − β 1 ) dW, v db = β 1 v db + (1 − β 1 ) db,
s dω = β 2 s dω + (1 − β 2 ) dW
2 , s db = β 2 s db + (1 − β 2 ) db
2 .
(2)
In these equations, β 1 and β 2 are gradient cumulative indexes, which are usually
0.9 and 0.999, respectively; dW and db are the gradients of the weight (W ) and
bias (b). Since the exponential moving average will cause a large difference from
the initial value in early iteration, gradient cumulates, and squared cumulates
need to be corrected by:
v
c
dω =
v dω
1 − β t
1
, v
c
db =
v db
1 − β t
1
, s
c
dω =
s dω
1 − β t
2
, s
c
db =
s db
1 − β t
2
.
(3)
In these equations, v
c
dω , v
c
db , s
c
dω and s
c
db are the corrections of v dω , v db , s dω and
s db respectively. W and b can be updated by:
W
(n) = W
(n−1)
− α
v
c
dω
s c
dω + ε
, b
(n) = b
(n−1)
− α
v
c
db
s c
db + ε
.
(4)
In these equations, ε is the smooth term, which is usually 10
−8 ; α is the learning
rate, which needs to be fine-tuned during training.
2.3 Classifier
In this paper, Softmax classifier, one of the most commonly used classifier in
CNN, is used as the classification module. Softmax classifier maps the output of
multiple neurons to the interval of (0, 1), which can be understood as a probability problem, given as follows:
p
y
(i) = j|x
(i) ; θ
=
e
θj x
(i)
k
l=1 e θ l x (i)
.
(5)
In this equation, θ j is the j
th weight vector; x
(i) is the ith data sample; y
(i)
is the output of the ith neuron; k is the total number of possible cases of the
classification. In this paper, cross entry is used as the loss function.
Y. Zhou et al.
weighted average for the gradient, which is beneficial to correct the swing amplitude and speed up the convergence. The Adam algorithm combines the Momentum algorithm and the RMSProp algorithm, thus, it has the advantages of both
algorithms.
At the beginning of the training, gradient cumulates (v dω and v db ) and
squared cumulates (s dω and s db ) are initialized as:
v dω = v db = s dω = s db = 0.
(1)
Assume that in the tth training, the parameter updates for Momentum and
RMSProp are calculated by:
v dω = β 1 v dω + (1 − β 1 ) dW, v db = β 1 v db + (1 − β 1 ) db,
s dω = β 2 s dω + (1 − β 2 ) dW
2 , s db = β 2 s db + (1 − β 2 ) db
2 .
(2)
In these equations, β 1 and β 2 are gradient cumulative indexes, which are usually
0.9 and 0.999, respectively; dW and db are the gradients of the weight (W ) and
bias (b). Since the exponential moving average will cause a large difference from
the initial value in early iteration, gradient cumulates, and squared cumulates
need to be corrected by:
v
c
dω =
v dω
1 − β t
1
, v
c
db =
v db
1 − β t
1
, s
c
dω =
s dω
1 − β t
2
, s
c
db =
s db
1 − β t
2
.
(3)
In these equations, v
c
dω , v
c
db , s
c
dω and s
c
db are the corrections of v dω , v db , s dω and
s db respectively. W and b can be updated by:
W
(n) = W
(n−1)
− α
v
c
dω
s c
dω + ε
, b
(n) = b
(n−1)
− α
v
c
db
s c
db + ε
.
(4)
In these equations, ε is the smooth term, which is usually 10
−8 ; α is the learning
rate, which needs to be fine-tuned during training.
2.3 Classifier
In this paper, Softmax classifier, one of the most commonly used classifier in
CNN, is used as the classification module. Softmax classifier maps the output of
multiple neurons to the interval of (0, 1), which can be understood as a probability problem, given as follows:
p
y
(i) = j|x
(i) ; θ
=
e
θj x
(i)
k
l=1 e θ l x (i)
.
(5)
In this equation, θ j is the j
th weight vector; x
(i) is the ith data sample; y
(i)
is the output of the ith neuron; k is the total number of possible cases of the
classification. In this paper, cross entry is used as the loss function.