114
Biomedical Signal and Image Processing
The 2-D definitions for all concepts described previously are straightforward.
In general, there are two approaches in biomedical signal and image processing.
In the first approach, every signal is assumed to be deterministic and as a result,
every measurement is directly used in Fourier, wavelet, and other types of analysis.
Obviously, while the assumption made in this approach is not very realistic, the
computational steps are simpler and straightforward. In the second approach, all
signals are assumed to be stochastic processes, and, therefore, instead of applying
the FT and other techniques directly on the measured signals, the secondary signals
formed by concepts such as mean, variance, autocorrelation, and cross-correlation
functions are used for Fourier and wavelet analysis. In this book, in order to cover
the basic ideas of both approaches, we cover the concepts and applications from both
approaches.
6.5 INTRODUCTION TO INFORMATION THEORY
Information theory is a field of study that investigates the mathematical formulation
of “information.” This theory is heavily used in signal and image processing, and as
result, some fundamental concepts of this theory are covered next.
6.5.1 ENTROPY
The basic definition in information theory, i.e., entropy, is designed to measure the
amount of information in a statement or variable. In order to reach to a suitable
definition for information, let us start with an intuitive comparison of the information contained in the following three statements: (1) Earth is spherical, (2) humans
have six legs, and (3) it is going to rain 40 days from today. The first statement, even
though very true, has no surprise in it, and we gain no information from it. Obviously,
no information is gained from the second statement, as we all know beforehand that
this statement is false. The third statement, on the other hand, makes a prediction
that may or may not be true.
From the preceding example, one can see that the basic measure of information is
the degree of “surprise”; if the statement has obviously true or obviously false claims
in it (e.g., 1 + 1 = 2 or 1 + 4 = 10), then there is no surprise or information in it. This
suggests that the concept of probability can be used to form a potential measure of
information. Assume that the probabilities of a random variable X with the outcomes
x i ’s are given as p i ’s, where i = 0, 1, 2,…, N − 1. Then one can suggest that the measure 1/p i could be a measure of information. For this measure, when the probability
of an incident gets smaller (and therefore we are more surprised), more information
is gained. This measure has two problems associated with it. First, for zero probability, the information is calculated to be infinity. This is not what we expected as the
information in an obviously wrong statement must be zero. The other problem deals
with the outcome whose probability is one. For such a case, the calculated information is also one while we expect zero information from an obviously true statement.
In order to address at least one of the aforementioned issues, we can modify our
information measure to “log(1/p i ).” This modification ensures that for p i = 1, the calculated information is indeed mapped to zero. However, the information calculated
Biomedical Signal and Image Processing
The 2-D definitions for all concepts described previously are straightforward.
In general, there are two approaches in biomedical signal and image processing.
In the first approach, every signal is assumed to be deterministic and as a result,
every measurement is directly used in Fourier, wavelet, and other types of analysis.
Obviously, while the assumption made in this approach is not very realistic, the
computational steps are simpler and straightforward. In the second approach, all
signals are assumed to be stochastic processes, and, therefore, instead of applying
the FT and other techniques directly on the measured signals, the secondary signals
formed by concepts such as mean, variance, autocorrelation, and cross-correlation
functions are used for Fourier and wavelet analysis. In this book, in order to cover
the basic ideas of both approaches, we cover the concepts and applications from both
approaches.
6.5 INTRODUCTION TO INFORMATION THEORY
Information theory is a field of study that investigates the mathematical formulation
of “information.” This theory is heavily used in signal and image processing, and as
result, some fundamental concepts of this theory are covered next.
6.5.1 ENTROPY
The basic definition in information theory, i.e., entropy, is designed to measure the
amount of information in a statement or variable. In order to reach to a suitable
definition for information, let us start with an intuitive comparison of the information contained in the following three statements: (1) Earth is spherical, (2) humans
have six legs, and (3) it is going to rain 40 days from today. The first statement, even
though very true, has no surprise in it, and we gain no information from it. Obviously,
no information is gained from the second statement, as we all know beforehand that
this statement is false. The third statement, on the other hand, makes a prediction
that may or may not be true.
From the preceding example, one can see that the basic measure of information is
the degree of “surprise”; if the statement has obviously true or obviously false claims
in it (e.g., 1 + 1 = 2 or 1 + 4 = 10), then there is no surprise or information in it. This
suggests that the concept of probability can be used to form a potential measure of
information. Assume that the probabilities of a random variable X with the outcomes
x i ’s are given as p i ’s, where i = 0, 1, 2,…, N − 1. Then one can suggest that the measure 1/p i could be a measure of information. For this measure, when the probability
of an incident gets smaller (and therefore we are more surprised), more information
is gained. This measure has two problems associated with it. First, for zero probability, the information is calculated to be infinity. This is not what we expected as the
information in an obviously wrong statement must be zero. The other problem deals
with the outcome whose probability is one. For such a case, the calculated information is also one while we expect zero information from an obviously true statement.
In order to address at least one of the aforementioned issues, we can modify our
information measure to “log(1/p i ).” This modification ensures that for p i = 1, the calculated information is indeed mapped to zero. However, the information calculated