48
3 Basics of Neural Networks
can be written as
|h l =
m
|m l (m|J l |h l−1 ) .
(3.59)
Furthermore, the “variation” of (3.59) with respect to the parameter J is calculated
as
δ|h l =
m
|m
l ( l |h l−1
We name this part G l
δJ l |h l−1 + J l δ|h l−1
= G l
δJ l |h l−1 + J l δ|h l−1
.
(3.60)
Also, using the notation
d = |d ,
(3.61)
with the activation function of the final layer as σ N ( = − log σ m (z), the
softmax cross entropy that appears in the error function is
E = =d|h N .
(3.62)
If we want to consider the mean square error, σ N can be anything, and we can write
E =
1
2
(|d − |h N
2
=
1
2
N − d|h N − d .
(3.63)
In any case, the variation on the output of the final layer is written with the bra
vector,
δE
δ|h N
=
N − d|
=: :δ 0 |.
(3.64)
We have named this 0 |. If we take the variation of the error function for all J l
instead of the final layer output, δE can be transformed as follows by using (3.60)
repeatedly:
δE =
δE
δ|h N
δ|h N = =δ 0 |
δ|h N
Transform this with (3.60)
=
=δ 0 |G N
Define this =::δ 1 |
δJ N |h N−1 + J N δ|h N−1