3.1 Error Function from Statistical Mechanics
41
Here, let us introduce the following function called the softmax function, which is
an extension of the sigmoid function,
σ I (X) =
e X I
J e X J
.
(3.21)
Using this softmax function, we can write the Boltzmann weight as
Q J (d
I
= 1|x) = σ I (Jx + J) .
(3.22)
As in the case with a single teaching signal, we define the model as
Q J (x, d) = Q J (d
I
|x)P (x).
(3.23)
Then we are to consider the minimization of the relative entropy
D KL (P ||Q J ) = −
x,d
P (x, d) log Q J (d|x) + (J -independent part)
(3.24)
and in particular,
First term in (3.24) ≈ −
i:data
1
#
log Q J (d[i] | x[i])
=
−1
#
i:data
4
I =1
d I [i] log σ I (Jx[i] + J) .
(3.25)
As before, we look at the expectation value of the I component of d,
I J,x[i] =
d
d I · Q J (d | x[i])
= Q J (d I = 1| x[i]) = σ I (Jx[i] + J) .
(3.26)
So we introduce the cross entropy 6
L(X, d) = −
4
I =1
d I log X I ,
(3.27)
6 It is essentially the same as (3.16). To make them exactly the same, set d = (d, 1 − d) in (3.16).
41
Here, let us introduce the following function called the softmax function, which is
an extension of the sigmoid function,
σ I (X) =
e X I
J e X J
.
(3.21)
Using this softmax function, we can write the Boltzmann weight as
Q J (d
I
= 1|x) = σ I (Jx + J) .
(3.22)
As in the case with a single teaching signal, we define the model as
Q J (x, d) = Q J (d
I
|x)P (x).
(3.23)
Then we are to consider the minimization of the relative entropy
D KL (P ||Q J ) = −
x,d
P (x, d) log Q J (d|x) + (J -independent part)
(3.24)
and in particular,
First term in (3.24) ≈ −
i:data
1
#
log Q J (d[i] | x[i])
=
−1
#
i:data
4
I =1
d I [i] log σ I (Jx[i] + J) .
(3.25)
As before, we look at the expectation value of the I component of d,
I J,x[i] =
d
d I · Q J (d | x[i])
= Q J (d I = 1| x[i]) = σ I (Jx[i] + J) .
(3.26)
So we introduce the cross entropy 6
L(X, d) = −
4
I =1
d I log X I ,
(3.27)
6 It is essentially the same as (3.16). To make them exactly the same, set d = (d, 1 − d) in (3.16).