64
4 Advanced Neural Networks
Backpropagation of error
Let us take a look at backpropagation of error in a recurrent neural network. We
use the following recursive expression repeatedly:
δ|h(t)
=
m
|m
•
x |x(t) + +m|J h |h(t − 1)
We name it =:G(t )
⎛
⎝
δJ x |x(t)
+δJ h |h(t − 1)
+J h δ|h(t − 1)
⎞
⎠
= G(t)δJ x |x(t) + G(t)δJ h |h(t − 1) + G(t)J h δ|h(t − 1) .
(4.14)
Error function L is typically
L = =d(1)|h(1) + +d(2)|h(2) + . . . ,
(4.15)
and it is just a summation of individual terms. So it would be sufficient to consider
the tth term:
δd(t)|h(t) = =d(t)|δ|h(t)
= =d(t)|G(t)
=::δ t (t)|
δJ x |x(t) + δJ h |h(t − 1) + J h δ|h(t − 1)
= =δ t (t)|δJ x |x(t) + +δ t (t)|δJ h |h(t − 1)
+ +δ t (t)|J h G(t − 1)
=::δ t (t−1)|
δJ x |x(t − 1) + δJ h |h(t − 2) + J h δ|h(t − 2)
= . . .
= =δ t (t)|δJ x |x(t) + +δ t (t)|δJ h |h(t − 1)
+ +δ t (t − 1)|δJ x |x(t − 1) + +δ t (t − 1)|δJ h |h(t − 2)
. . .
+ +δ t (1)|δJ x |x(1) + +δ t (1)|δJ h |h(0)
(4.16)
Here, using
δJ =
m,n
|m
mn ,
(4.17)