116
4: Stefan A. Robila
In this case, the mutual information of the components of u can be expressed
as the difference between the sum of marginal (individual) entropies and the
joint entropy:
m
I(ul. ... , u m } = L. H(Ui} - H(u} .
( 4.15)
i=1
The solution of the leA problem can be obtained by minimizing the mutual
information and making it as close to zero as possible. Thus, we define the
minimal mutual information (MMI) problem as follows. For an n dimensional
random vector x, find a pair (u, W), where u is an m dimensional random
vector, and W is an m x n matrix, such that:
u= Wx
(4.16)
and
I (UI. ... , um) = min {I (VI. ... , vm) Iv = Vx} ,
( 4.17)
where V is a m x n matrix.
The MMI problem can be used to obtain the solution for the leA problem.
From (4.1), if (5,A) is the solution to the leA problem, under the assumption
that A is invertible, we have:
( 4.18)
where the components of 5 are independent. The MMI problem defined by (4.16)
is identical to the leA problem if 5 and A -1 are set equal to u and W, respectively. Since the components of 5 are independent, the mutual information is
zero, i. e.:
( 4.19)
In this case, 5 satisfies the second condition for the MMI solution (4.17).
Hence, the leA solution (5, A -1) can be obtained by solving the MMI problem.
The equivalence between the leA and the MMI problems allows the development of a practical procedure to solve the leA problem. In other words, the
leA solution is found by minimizing the mutual information. If MI reaches
the minimum achievable value of zero, complete independence of the components is achieved. Otherwise, the components are as independent as possible.
This will occur in situations where complete independence is not achievable
(Hyvarinen et al. 2001; Lee 1998).
Let us now develop an algorithm for the solution of the MMI problem. The
goal is to determine the gradient of the mutual information with respect to the
elements of W. Once the gradient is computed, it is used as the iterative step
for updating the elements of W in the gradient-based optimization algorithm:
aI(ul. ... ,um )
W = W + ..1 W = W - -'--------"aw
(4.20)