error system akin to repeating a maze periodically until the optimal route has been
determined. The ANN is not given a specific path to take, and must be tuned through
numerous training cycles [19].
Recurrent neural networks (RNNs) are a sequence based model of ANNs. RNN’s
can be separated into two primary classes, the RNN will either be presented with a
constant, single input and stabilize to a desired state or presented with variable inputs
fluctuating over time with the intention of yielding time-dependent outputs
[20]. RNN’s are capable of using their internal memory to process input sequences
allowing for complex applications including speech and handwriting recognition
[20, 21].
Convolutional neural networks (CNNs) function similar to a general ANN,
however neurons are now arranged in three dimensions and characterized by height,
width and depth. This manipulates the manner in which patterns are constrained and
scales more efficiently than a typical ANN for larger pattern recognition systems.
CNNs utilize three core layers: the convolutional layer, pooling layer and fullyconnected layer (this is present in a typical ANN). The convolutional layer computes
output neurons in relation to a dot product between the input region and its
corresponding weights. After the convolutional layer, an element based activation
function is applied followed by the pooling layer which down samples the width and
height dimensions while leaving the CNN volume intact; both of these act as fixed
pre-determined functions. Finally, the fully-connected layer will compute class
scores which relate the pattern to its desired category. Training through
backpropagation is only necessary for the convolutional and fully-connected layers.
CNNs core design achieves much more efficient pattern classification than a regular
ANN and makes it readily implementable in deep neural networks for image and
speech recognition.
Recursive neural networks (rNNs) are another model applicable to deep learning
that function by recursively applying an identical set of weights to a given system.
This achieves structured predictions based on variable inputs; this architecture also
allows for the flow of data in any given direction. While CNNs are good deep
learning models for pattern recognition, rNNs offer the ability to predict variations in
hierarchical systems, demonstrated in natural language processing, and can be
regarded as a linear modification of RNNs though are unable to parse tree-like
hierarchies [20].
Multi-layer perceptron’s (MLPs) are a class of ANNs which implement feed
forward loops. All hidden and output layers in the MLP act as a neuron responding to
stimuli (inputs) which rely on a non-linear, typically sigmoidal or hyperbolic
tangent, activation function. The activation function acts as a weighted tolerance
value governing the output function. MLPs typically utilize backpropagation algorithms for training and the weights can be updated iteratively after each testing phase
(often results in chaotic alterations) or in unison after all weights have been analyzed
(batch learning, often yields more stable alterations) [22]. By defining classes to
analyze, MLPs can be applied to many different tasks including 3D image recognition and handwritten image recognition.
206
R. Aguilera et al.
determined. The ANN is not given a specific path to take, and must be tuned through
numerous training cycles [19].
Recurrent neural networks (RNNs) are a sequence based model of ANNs. RNN’s
can be separated into two primary classes, the RNN will either be presented with a
constant, single input and stabilize to a desired state or presented with variable inputs
fluctuating over time with the intention of yielding time-dependent outputs
[20]. RNN’s are capable of using their internal memory to process input sequences
allowing for complex applications including speech and handwriting recognition
[20, 21].
Convolutional neural networks (CNNs) function similar to a general ANN,
however neurons are now arranged in three dimensions and characterized by height,
width and depth. This manipulates the manner in which patterns are constrained and
scales more efficiently than a typical ANN for larger pattern recognition systems.
CNNs utilize three core layers: the convolutional layer, pooling layer and fullyconnected layer (this is present in a typical ANN). The convolutional layer computes
output neurons in relation to a dot product between the input region and its
corresponding weights. After the convolutional layer, an element based activation
function is applied followed by the pooling layer which down samples the width and
height dimensions while leaving the CNN volume intact; both of these act as fixed
pre-determined functions. Finally, the fully-connected layer will compute class
scores which relate the pattern to its desired category. Training through
backpropagation is only necessary for the convolutional and fully-connected layers.
CNNs core design achieves much more efficient pattern classification than a regular
ANN and makes it readily implementable in deep neural networks for image and
speech recognition.
Recursive neural networks (rNNs) are another model applicable to deep learning
that function by recursively applying an identical set of weights to a given system.
This achieves structured predictions based on variable inputs; this architecture also
allows for the flow of data in any given direction. While CNNs are good deep
learning models for pattern recognition, rNNs offer the ability to predict variations in
hierarchical systems, demonstrated in natural language processing, and can be
regarded as a linear modification of RNNs though are unable to parse tree-like
hierarchies [20].
Multi-layer perceptron’s (MLPs) are a class of ANNs which implement feed
forward loops. All hidden and output layers in the MLP act as a neuron responding to
stimuli (inputs) which rely on a non-linear, typically sigmoidal or hyperbolic
tangent, activation function. The activation function acts as a weighted tolerance
value governing the output function. MLPs typically utilize backpropagation algorithms for training and the weights can be updated iteratively after each testing phase
(often results in chaotic alterations) or in unison after all weights have been analyzed
(batch learning, often yields more stable alterations) [22]. By defining classes to
analyze, MLPs can be applied to many different tasks including 3D image recognition and handwritten image recognition.
206
R. Aguilera et al.