112
6 Unsupervised Deep Learning
expected that creating a theory of why it works will require incorporating conditions
not only about the algorithm but also about the structure of the actual network used.
As of April 2019, to the best of the authors’ knowledge, the theoretical guarantees
of the (6.48) version of GANs are an open question.
6.3.1 Energy-Based GAN
Another example of V G,D is called an energy-based GAN [81], which is defined as
V D (G, D) = =D(x) x∼P (x) + +max(0, m − D(x)) x∼Q G (x) ,
(6.49)
V G (G, D) = =D(x) x∼Q G (x) .
(6.50)
Here m(> 0) is a manually set value that corresponds to the energy of the fake data.
It is the energy to measure “how much data x looks real,” which is modeled by
the classifier D. As in an ordinary physical system, energy must be bounded from
below:
D(x) ≥ 0 .
(6.51)
In the energy-based GAN, a classifier network is configured to satisfy this energy
condition (6.51). Under these settings, The training of D is performed as follows:
If x is real, make D(x) close to 0.
If x is fake, make D(x) close to m.
(6.52)
This corresponds to making the value of (6.49) as small as possible. 8 The training
of G is to make x fake = G(z) have the lowest possible energy (= realistic). The
“equilibrium point” (6.39) and (6.40) based on (6.49) and (6.50) can be shown to
give Q G ∗ (x) = P (x) without the help of the minimax theorem. The proof might be
too mathematical, but here we shall show it. 9
Property obtained from equilibrium point of V D
The inequality (6.40) says that for V D (G ∗ , D) into which G = G ∗ is substituted,
D ∗ takes the minimum value. So we start by looking at what happens to D(x) that
minimizes the following:
V D (G
∗ , D) =
dx
P (x)D(x) + Q G ∗ (x) max(0, m − D(x))
.
(6.53)
8 Just as V D in the original GAN corresponds to the cross entropy, the objective function (6.49)
called hinge loss corresponds to the error function of a support vector machine (which is not
described in this book).
9 This is a detailed version of the proof provided in the appendix of the original paper.
6 Unsupervised Deep Learning
expected that creating a theory of why it works will require incorporating conditions
not only about the algorithm but also about the structure of the actual network used.
As of April 2019, to the best of the authors’ knowledge, the theoretical guarantees
of the (6.48) version of GANs are an open question.
6.3.1 Energy-Based GAN
Another example of V G,D is called an energy-based GAN [81], which is defined as
V D (G, D) = =D(x) x∼P (x) + +max(0, m − D(x)) x∼Q G (x) ,
(6.49)
V G (G, D) = =D(x) x∼Q G (x) .
(6.50)
Here m(> 0) is a manually set value that corresponds to the energy of the fake data.
It is the energy to measure “how much data x looks real,” which is modeled by
the classifier D. As in an ordinary physical system, energy must be bounded from
below:
D(x) ≥ 0 .
(6.51)
In the energy-based GAN, a classifier network is configured to satisfy this energy
condition (6.51). Under these settings, The training of D is performed as follows:
If x is real, make D(x) close to 0.
If x is fake, make D(x) close to m.
(6.52)
This corresponds to making the value of (6.49) as small as possible. 8 The training
of G is to make x fake = G(z) have the lowest possible energy (= realistic). The
“equilibrium point” (6.39) and (6.40) based on (6.49) and (6.50) can be shown to
give Q G ∗ (x) = P (x) without the help of the minimax theorem. The proof might be
too mathematical, but here we shall show it. 9
Property obtained from equilibrium point of V D
The inequality (6.40) says that for V D (G ∗ , D) into which G = G ∗ is substituted,
D ∗ takes the minimum value. So we start by looking at what happens to D(x) that
minimizes the following:
V D (G
∗ , D) =
dx
P (x)D(x) + Q G ∗ (x) max(0, m − D(x))
.
(6.53)
8 Just as V D in the original GAN corresponds to the cross entropy, the objective function (6.49)
called hinge loss corresponds to the error function of a support vector machine (which is not
described in this book).
9 This is a detailed version of the proof provided in the appendix of the original paper.