146
Biomedical Signal and Image Processing
A multilayer sigmoid neural network always has one input layer (X units) that
has as many neurons as there are inputs to the network. Similarly, there is always an
output layer (Y or T units) that has as many outputs as the number of outputs of the
networks. Such structures can have an arbitrary number of hidden layers (Z units)
between the input and output layers. Each of these hidden layers can have an arbitrary number of neurons. The number of hidden layers and the number of neurons in
each layer are to the most part chosen experimentally through a process of trial and
error. Even though paradigms such as computational learning theory have attempted
to create rules for setting the optimal number of hidden layers and the number of
neurons in each hidden layer, these numbers are often set in such a way to avoid the
overcomplexity of the network and, at the same time, provide a powerful function to
estimate complex problems.
Backpropagation neural networks, another name given to multilayer sigmoid neural networks, are among the most popular types of the neural networks used in real
applications. In reality, backpropagation as defined later is merely one method of
training sigmoid neural networks, but since this algorithm is almost always used for
training these structures, in many books, multilayer sigmoid neural networks are
also called backpropagation neural networks. Another common name that is often
mistakenly used to describe multilayer sigmoid neural networks is multilayer perceptrons. As we will see later on, the type of neurons used in multilayer sigmoid
neural networks are continuous-output neurons and not perceptrons that are discrete
neurons.
7.7.2.1 Activation Function
In perceptron, the activation function was a simple thresholding process, i.e., if the
weighted sum of input is less than a certain threshold value, the output is set to some
value, and if the weighted sum is less than the threshold, a different value is introduced as the output of the neuron. This threshold effect simply makes the perceptron
a discrete neuron, i.e., the output value is not continuous and belongs to a discrete set.
As briefly mentioned before, the activation functions and neurons used in multilayer
sigmoid neural networks are continuous, differentiable, and monotonically increasing. These characteristics for the activation functions are needed to guarantee that
the network and its elements (e.g., neurons outputs) are all differentiable. As we will
see in this section, almost all training methods used for multilayer sigmoid neural
networks apply partial derivatives of the neurons, layers, and final output. In addition, it is often preferred to have the activation functions whose derivates can be
computed easily.
One of the most popular activation functions is binary sigmoid function (also
referred to as “logsig”):
1
f x
1 ( ) =
(7.21)
1 + exp( − x )
which gives the derivate f x
1 ′( ) as follows:
f x
1 ′ ( ) = f x
1 ( )[1 − f x
1 ( )]
(7.22)
Biomedical Signal and Image Processing
A multilayer sigmoid neural network always has one input layer (X units) that
has as many neurons as there are inputs to the network. Similarly, there is always an
output layer (Y or T units) that has as many outputs as the number of outputs of the
networks. Such structures can have an arbitrary number of hidden layers (Z units)
between the input and output layers. Each of these hidden layers can have an arbitrary number of neurons. The number of hidden layers and the number of neurons in
each layer are to the most part chosen experimentally through a process of trial and
error. Even though paradigms such as computational learning theory have attempted
to create rules for setting the optimal number of hidden layers and the number of
neurons in each hidden layer, these numbers are often set in such a way to avoid the
overcomplexity of the network and, at the same time, provide a powerful function to
estimate complex problems.
Backpropagation neural networks, another name given to multilayer sigmoid neural networks, are among the most popular types of the neural networks used in real
applications. In reality, backpropagation as defined later is merely one method of
training sigmoid neural networks, but since this algorithm is almost always used for
training these structures, in many books, multilayer sigmoid neural networks are
also called backpropagation neural networks. Another common name that is often
mistakenly used to describe multilayer sigmoid neural networks is multilayer perceptrons. As we will see later on, the type of neurons used in multilayer sigmoid
neural networks are continuous-output neurons and not perceptrons that are discrete
neurons.
7.7.2.1 Activation Function
In perceptron, the activation function was a simple thresholding process, i.e., if the
weighted sum of input is less than a certain threshold value, the output is set to some
value, and if the weighted sum is less than the threshold, a different value is introduced as the output of the neuron. This threshold effect simply makes the perceptron
a discrete neuron, i.e., the output value is not continuous and belongs to a discrete set.
As briefly mentioned before, the activation functions and neurons used in multilayer
sigmoid neural networks are continuous, differentiable, and monotonically increasing. These characteristics for the activation functions are needed to guarantee that
the network and its elements (e.g., neurons outputs) are all differentiable. As we will
see in this section, almost all training methods used for multilayer sigmoid neural
networks apply partial derivatives of the neurons, layers, and final output. In addition, it is often preferred to have the activation functions whose derivates can be
computed easily.
One of the most popular activation functions is binary sigmoid function (also
referred to as “logsig”):
1
f x
1 ( ) =
(7.21)
1 + exp( − x )
which gives the derivate f x
1 ′( ) as follows:
f x
1 ′ ( ) = f x
1 ( )[1 − f x
1 ( )]
(7.22)