Independent Component Analysis
121
If, for all i, the derivative off; (Ui) with respect to Ui matches the probability
density function of Ui" i. e.
( .) _ of; (Ui)
p Ut - - : 1 - - '
uUi
(4.38)
then all the marginal entropies become zero and
I (n, .. . ,Yn) = -H (y) .
(4.39)
Thus, maximization of the joint entropy becomes equivalent to the minimization of the mutual information. If the nonlinearities do not match the
probability density functions, then the algorithm may introduce errors in the
estimation of the independent components.
The gradient-based iterative step for the Infomax algorithm (Bell and Sejnowski 1995) is given as
(4.40)
which is identical to the one in (4.31) with the observation that gi(.) are the
derivatives off;(·).
4.3.3
ICA Solution through Non-Gaussianity Maximization
A different approach for the solution of the ICA problem is to find a linear
transform such that the resulting components are uncorrelated and as nonGaussian as possible (Hyvarinen et al. 2001).
According to the central limit theorem (Hyvarinen et al. 2001), the distribution of any linear mixture of non-Gaussian independent components tends
to be closer to the Gaussian distribution than the distributions of the initial
components. In the effort to recover the independent components from the
observed vector x, we are trying to find the transform W such that:
u = Wx= WAs
(4.41)
and that the components of u are as independent as possible. In (4.41), s is
the vector of original independent components, and x = As is the vector of
observed components.
WandA are linear transforms, so the components of u will be more Gaussian
than the components of s. Since we assume that the components of s are nonGaussian and independent, the maximum non-Gaussianity for u is achieved
when the components of u are equal to the components of s, i. e. when:
(4.42)
121
If, for all i, the derivative off; (Ui) with respect to Ui matches the probability
density function of Ui" i. e.
( .) _ of; (Ui)
p Ut - - : 1 - - '
uUi
(4.38)
then all the marginal entropies become zero and
I (n, .. . ,Yn) = -H (y) .
(4.39)
Thus, maximization of the joint entropy becomes equivalent to the minimization of the mutual information. If the nonlinearities do not match the
probability density functions, then the algorithm may introduce errors in the
estimation of the independent components.
The gradient-based iterative step for the Infomax algorithm (Bell and Sejnowski 1995) is given as
(4.40)
which is identical to the one in (4.31) with the observation that gi(.) are the
derivatives off;(·).
4.3.3
ICA Solution through Non-Gaussianity Maximization
A different approach for the solution of the ICA problem is to find a linear
transform such that the resulting components are uncorrelated and as nonGaussian as possible (Hyvarinen et al. 2001).
According to the central limit theorem (Hyvarinen et al. 2001), the distribution of any linear mixture of non-Gaussian independent components tends
to be closer to the Gaussian distribution than the distributions of the initial
components. In the effort to recover the independent components from the
observed vector x, we are trying to find the transform W such that:
u = Wx= WAs
(4.41)
and that the components of u are as independent as possible. In (4.41), s is
the vector of original independent components, and x = As is the vector of
observed components.
WandA are linear transforms, so the components of u will be more Gaussian
than the components of s. Since we assume that the components of s are nonGaussian and independent, the maximum non-Gaussianity for u is achieved
when the components of u are equal to the components of s, i. e. when:
(4.42)