8 Chain Rule Optimal Transport
193
convex, and introduce its relaxed Sinkhorn distance (SCROT) for fast estimation.
Numerical experiments in Sect. 8.4 highlight quantitatively the upper bound performance of the (S)CROT distances for bounding the total variation distance, the
Wasserstein W p metric, and the Rényi α-divergences.
• We design a novel learning algorithm for GMMs by simplifying KDEs with
respect to SCROT that yields in that case a closed-form formula (Eq. 8.15) in
Sect. 8.5, and demonstrate experimentally better results than the ExpectationMaximization (EM) implementation [11] in sklearn [49] on MNIST [32] and
Fashion MNIST [64] datasets.
8.2 Chain Rule Optimal Transport
Recall the basic chain rule factorization of a joint probability distribution:
p(x, y) = p(y) p(x|y),
where probability p(y) is the marginal probability, and probability p(x|y) is the conditional probability. Given p(y) and p(x|y) in certain families of simple probability
distributions, one can get a density model through marginalization:
p(x) =
p(x, y)dy.
For example, for latent models like statistical mixtures or hidden Markov models [58,
65], x plays the role of the observed variable while y denotes the hidden variable [16]
(unobserved so that inference has to tackle incomplete data, say, using the EM algorithm [11]. Let X = {p(x)} and Y = {p(y)} denote the manifolds of marginal probability densities; let C = {p(x|y)} denote the manifold of conditional probability
density. We state the generic definition of the Chain Rule Optimal Transport (CROT)
distance between the distributions p(x) and q(x) (with q(x) =
q(y)q(x|y)dy) as
follows:
Definition 8.1 (CROT distance) Given two multivariate distributions p(x, y) and
q(x, y), we define the Chain Rule Optimal Transport as follows:
H D ( p, q):=
inf
r ∈( p(y),q(z))
E r (y,z)
D
p(x|y), q(x|z)
,
(8.1)
where D(·, ·) is a ground distance defined on conditional density manifold C =
{ p(x|y)} (e.g., the Total Variation), ( p(y), q(z)) is the set of all probability measures on Y
2 satisfying the constraints
r (y, z)dz = p(y) and
r (y, z)dy = q(z),
and E r (y,z) denotes the expectation with respect to r (y, z).
193
convex, and introduce its relaxed Sinkhorn distance (SCROT) for fast estimation.
Numerical experiments in Sect. 8.4 highlight quantitatively the upper bound performance of the (S)CROT distances for bounding the total variation distance, the
Wasserstein W p metric, and the Rényi α-divergences.
• We design a novel learning algorithm for GMMs by simplifying KDEs with
respect to SCROT that yields in that case a closed-form formula (Eq. 8.15) in
Sect. 8.5, and demonstrate experimentally better results than the ExpectationMaximization (EM) implementation [11] in sklearn [49] on MNIST [32] and
Fashion MNIST [64] datasets.
8.2 Chain Rule Optimal Transport
Recall the basic chain rule factorization of a joint probability distribution:
p(x, y) = p(y) p(x|y),
where probability p(y) is the marginal probability, and probability p(x|y) is the conditional probability. Given p(y) and p(x|y) in certain families of simple probability
distributions, one can get a density model through marginalization:
p(x) =
p(x, y)dy.
For example, for latent models like statistical mixtures or hidden Markov models [58,
65], x plays the role of the observed variable while y denotes the hidden variable [16]
(unobserved so that inference has to tackle incomplete data, say, using the EM algorithm [11]. Let X = {p(x)} and Y = {p(y)} denote the manifolds of marginal probability densities; let C = {p(x|y)} denote the manifold of conditional probability
density. We state the generic definition of the Chain Rule Optimal Transport (CROT)
distance between the distributions p(x) and q(x) (with q(x) =
q(y)q(x|y)dy) as
follows:
Definition 8.1 (CROT distance) Given two multivariate distributions p(x, y) and
q(x, y), we define the Chain Rule Optimal Transport as follows:
H D ( p, q):=
inf
r ∈( p(y),q(z))
E r (y,z)
D
p(x|y), q(x|z)
,
(8.1)
where D(·, ·) is a ground distance defined on conditional density manifold C =
{ p(x|y)} (e.g., the Total Variation), ( p(y), q(z)) is the set of all probability measures on Y
2 satisfying the constraints
r (y, z)dz = p(y) and
r (y, z)dy = q(z),
and E r (y,z) denotes the expectation with respect to r (y, z).