6.3 Generative Adversarial Network
117
Generalization to Helmholtz free energy
The interesting thing about this distance is that there is an equivalent expression
which looks completely different. In order to derive it, once we put this system
in the nonzero temperature, and consider, not the minimum value of the internal
energy, but the minimum value of Helmholtz free energy. 13
D
T
W (P , Q) = min
π∈ (6.77)
U(π) − T S(π)
.
(6.79)
Here, we adopt the following entropy 14 of the transportation plan π,
S(π) = =− log π(x, y) + 1 (x,y)∼π(x,y) .
(6.80)
Because of the temperature, this does not actually satisfy the property of distance (6.78). However, in thermodynamics, free energy often gives a better perspective than internal energy, and accordingly it acquires good properties. 15
Now, we want to determine the transportation plan π that minimizes this free
energy, and a difficulty is how to take into account the constraint (6.77). We put
the constraint into the optimization by using the Lagrange multiplier method. The
expectation value is written in the form of an integral as
D
T
W (P , Q) = min
π
max
f,g
dxdy π(x, y)
E(x, y) + T log π(x, y) − T
+
dx f (x)
P (x) −
dy π(x, y)
+
dy g(y)
Q(y) −
dx π(x, y)
.
(6.81)
We consider changing the order, as min π max f,g = max f,g min π . To find the
minimum value over π first, it is sufficient to take the variation of π and set it
to zero,
0 = E(x, y) + T log π
∗ (x, y) − f (x) − g(y) .
(6.82)
Then we can substitute this solution and consider max f,g , to find another expression
D
T
W (P , Q) = max
f,g
f (x) x∼P (x) + +g(y) y∼Q(y) − T
dxdy π
∗ (x, y)
.
(6.83)
13 This generalization is not necessary to derive the final form of the WGAN, but considering the
Helmholtz free energy makes the derivation easier to understand.
14 The term “+1” is not necessary but it will make the later discussion cleaner.
15 For example, there is a way to reduce the amount of calculation in the actual calculation
algorithm compared to the original zero temperature problem [85].
117
Generalization to Helmholtz free energy
The interesting thing about this distance is that there is an equivalent expression
which looks completely different. In order to derive it, once we put this system
in the nonzero temperature, and consider, not the minimum value of the internal
energy, but the minimum value of Helmholtz free energy. 13
D
T
W (P , Q) = min
π∈ (6.77)
U(π) − T S(π)
.
(6.79)
Here, we adopt the following entropy 14 of the transportation plan π,
S(π) = =− log π(x, y) + 1 (x,y)∼π(x,y) .
(6.80)
Because of the temperature, this does not actually satisfy the property of distance (6.78). However, in thermodynamics, free energy often gives a better perspective than internal energy, and accordingly it acquires good properties. 15
Now, we want to determine the transportation plan π that minimizes this free
energy, and a difficulty is how to take into account the constraint (6.77). We put
the constraint into the optimization by using the Lagrange multiplier method. The
expectation value is written in the form of an integral as
D
T
W (P , Q) = min
π
max
f,g
dxdy π(x, y)
E(x, y) + T log π(x, y) − T
+
dx f (x)
P (x) −
dy π(x, y)
+
dy g(y)
Q(y) −
dx π(x, y)
.
(6.81)
We consider changing the order, as min π max f,g = max f,g min π . To find the
minimum value over π first, it is sufficient to take the variation of π and set it
to zero,
0 = E(x, y) + T log π
∗ (x, y) − f (x) − g(y) .
(6.82)
Then we can substitute this solution and consider max f,g , to find another expression
D
T
W (P , Q) = max
f,g
f (x) x∼P (x) + +g(y) y∼Q(y) − T
dxdy π
∗ (x, y)
.
(6.83)
13 This generalization is not necessary to derive the final form of the WGAN, but considering the
Helmholtz free energy makes the derivation easier to understand.
14 The term “+1” is not necessary but it will make the later discussion cleaner.
15 For example, there is a way to reduce the amount of calculation in the actual calculation
algorithm compared to the original zero temperature problem [85].