120
4: Stefan A. Robila
MMI algorithm converges very close to an leA solution. This is very important since the use of a single nonlinearity implies that the estimation of the
probability density functions does not have to be very precise. The success of
using only one function can be explained by the fact that a probability density
function mismatch may still lead to a solution that is sufficient for our purposes because its corresponding independent sources are scaled versions of
the initial independent sources. It is, however, important to indicate that not
all functions are equal. For example, a super-Gaussian-like function may not
model a sub-Gaussian source correctly. Therefore, remedial approaches that
involve prediction of the nature of the sources along with a switch between
sub-Gaussian and super-Gaussian functions have been proposed (Lee 1998).
Algorithms, other than the MMI algorithm have also been proposed. One
such algorithm is the Information Maximization leA (Infomax) algorithm
presented in Bell and Sejnowski (1995). While the update formula is identical
to the one generated through MMI, the setting for the Infomax algorithm is
slightly different. Following the transformation through W, the random vector
u is further processed through a set of nonlinear functionsf;(·) (see Fig. 4.3).
If these functions are invertible, the independence of the components of y is
equivalent to the independence of the components of u.
To understand how the nonlinearity can be used, it is interesting to look at
the mutual information for the components of y (Bell and Sejnowski 1995):
n
n
I(yJ,,,·,Yn) = -H(y) + LH(Yi) = -H(y) + LE {log (p (Yi)) I
i=!
i=!
= -H (y) - t. +Og (P(U;) tf~~~;) ) I (4.36)
since, according to Popoulis (1991), for a random variable v and a differentiable
function h(·):
(
/
Jh(V))
p(h(v)) = p(v)
a;- .
x
Xl---~
X2 ---~
Xn ---~
(4.37)
u
f(.)
y
U" ~ Yn
Fig. 4.3. Infomax algorithm for leA solution. Following the multiplication with W, the data
is further transformed through nonlinear functions. Independence of the components of u
is equivalent to the independence of the components of y. Infomax algorithm proceeds to
modify W such that components of y become independent
4: Stefan A. Robila
MMI algorithm converges very close to an leA solution. This is very important since the use of a single nonlinearity implies that the estimation of the
probability density functions does not have to be very precise. The success of
using only one function can be explained by the fact that a probability density
function mismatch may still lead to a solution that is sufficient for our purposes because its corresponding independent sources are scaled versions of
the initial independent sources. It is, however, important to indicate that not
all functions are equal. For example, a super-Gaussian-like function may not
model a sub-Gaussian source correctly. Therefore, remedial approaches that
involve prediction of the nature of the sources along with a switch between
sub-Gaussian and super-Gaussian functions have been proposed (Lee 1998).
Algorithms, other than the MMI algorithm have also been proposed. One
such algorithm is the Information Maximization leA (Infomax) algorithm
presented in Bell and Sejnowski (1995). While the update formula is identical
to the one generated through MMI, the setting for the Infomax algorithm is
slightly different. Following the transformation through W, the random vector
u is further processed through a set of nonlinear functionsf;(·) (see Fig. 4.3).
If these functions are invertible, the independence of the components of y is
equivalent to the independence of the components of u.
To understand how the nonlinearity can be used, it is interesting to look at
the mutual information for the components of y (Bell and Sejnowski 1995):
n
n
I(yJ,,,·,Yn) = -H(y) + LH(Yi) = -H(y) + LE {log (p (Yi)) I
i=!
i=!
= -H (y) - t. +Og (P(U;) tf~~~;) ) I (4.36)
since, according to Popoulis (1991), for a random variable v and a differentiable
function h(·):
(
/
Jh(V))
p(h(v)) = p(v)
a;- .
x
Xl---~
X2 ---~
Xn ---~
(4.37)
u
f(.)
y
U" ~ Yn
Fig. 4.3. Infomax algorithm for leA solution. Following the multiplication with W, the data
is further transformed through nonlinear functions. Independence of the components of u
is equivalent to the independence of the components of y. Infomax algorithm proceeds to
modify W such that components of y become independent