118
6 Unsupervised Deep Learning
Such a situation that the original problem (min) becomes equivalent to some other
problem (max) often appears in machine learning, 16 and it is called a strong duality
in optimization problems. Looking at the derivation here, we see that the new
degree of freedom of the dual problem is the Lagrange multiplier that expresses the
constraint, and it is similar to the duality in statistical mechanics and field theories.
At the time of writing this, the authors cannot tell if this is just a similarity or has a
profound meaning.
Kantorovich-Rubinstein duality
Consider taking a zero temperature limit T → +0 to return to the original
problem (6.76). The dual problem appears to be able to reach the limit without
any difficulty. Solving (6.82) gives
π
∗ (x, y) = e
−H (x,y)
T
,
(6.84)
H (x, y) = E(x, y) − f (x) − g(y).
(6.85)
Since (6.83) is about the maximum value, from the beginning, it is better to remove
f, g with which the last term becomes −∞. According to this argument, we find a
condition similar to (6.51), where the “Hamiltonian” (6.85) is bounded from below,
H (x, y) ≥ 0, i.e., f (x) + g(y) ≤ E(x, y) .
(6.86)
Therefore,
D W (P , Q) =
max
f (x)+g(y)≤E(x,y)
f (x) x∼P (x) + +g(y) y∼Q(y)
.
(6.87)
In particular, if the energy function for transport is taken as E(x, y) = ||x − y||, at
least g = −f must be attained to achieve the maximum, so finally
D W (P , Q) =
max
f (x)−f (y)≤||x−y||
f (x) x∼P (x) − −f (y) y∼Q(y)
.
(6.88)
In other words, the point is that f is restricted to functions with Lipschitz continuity.
WGAN
Incorporating idea of GAN into this duality leads to the idea of Wasserstein GAN.
Simply we set Q = Q G and optimize G to minimize the Wasserstein distance:
min
G
D W (P , Q G ) = min
G
max
f (x)−f (y)≤||x−y||
f (x) x∼P (x) − −f (y) y∼Q(y)
.
(6.89)
16 As another example, support vector machines (which are not described in this book) have also a
duality.
6 Unsupervised Deep Learning
Such a situation that the original problem (min) becomes equivalent to some other
problem (max) often appears in machine learning, 16 and it is called a strong duality
in optimization problems. Looking at the derivation here, we see that the new
degree of freedom of the dual problem is the Lagrange multiplier that expresses the
constraint, and it is similar to the duality in statistical mechanics and field theories.
At the time of writing this, the authors cannot tell if this is just a similarity or has a
profound meaning.
Kantorovich-Rubinstein duality
Consider taking a zero temperature limit T → +0 to return to the original
problem (6.76). The dual problem appears to be able to reach the limit without
any difficulty. Solving (6.82) gives
π
∗ (x, y) = e
−H (x,y)
T
,
(6.84)
H (x, y) = E(x, y) − f (x) − g(y).
(6.85)
Since (6.83) is about the maximum value, from the beginning, it is better to remove
f, g with which the last term becomes −∞. According to this argument, we find a
condition similar to (6.51), where the “Hamiltonian” (6.85) is bounded from below,
H (x, y) ≥ 0, i.e., f (x) + g(y) ≤ E(x, y) .
(6.86)
Therefore,
D W (P , Q) =
max
f (x)+g(y)≤E(x,y)
f (x) x∼P (x) + +g(y) y∼Q(y)
.
(6.87)
In particular, if the energy function for transport is taken as E(x, y) = ||x − y||, at
least g = −f must be attained to achieve the maximum, so finally
D W (P , Q) =
max
f (x)−f (y)≤||x−y||
f (x) x∼P (x) − −f (y) y∼Q(y)
.
(6.88)
In other words, the point is that f is restricted to functions with Lipschitz continuity.
WGAN
Incorporating idea of GAN into this duality leads to the idea of Wasserstein GAN.
Simply we set Q = Q G and optimize G to minimize the Wasserstein distance:
min
G
D W (P , Q G ) = min
G
max
f (x)−f (y)≤||x−y||
f (x) x∼P (x) − −f (y) y∼Q(y)
.
(6.89)
16 As another example, support vector machines (which are not described in this book) have also a
duality.