2.3 Deep Learning
Modern neuromorphic system design strives to build the same adaptive learning
capacity found in biological neural systems into an electronic device. One way to
incorporate this system design is through the employment of machine learning
algorithms. Machine learning algorithms attempt to emulate neuronal design in
software. The learning process can be accomplished in a supervised or unsupervised
manner, with the key distinction being whether the data fed into the algorithm is
labeled (supervised) or unlabeled (unsupervised) [23]. Presently, supervised learning
is the most common form of machine learning, regardless of the algorithm
employed.
Historically, machine learning algorithms and techniques struggled to process
natural data sets in raw form and only through extensive work by engineers could
machine learning be tailored to perform tasks such as speech recognition [23]. This
has changed with the development of new methods, feature learning or representation learning, which are capable of identifying the minimum features that represents
each class of object for detection and/or classification within a data set. Those
methods led to the creation of the well-publicized machine learning technique
known as deep-learning that simulates a multi-layered neuron architecture. Its
purpose is for organizing unstructured data and applying this learned information
to various tasks, such as computer vision, speech recognition and bioinformatics. In
the case of supervised learning, the process starts by amassing a large data set where
each element of the data set is labeled with its corresponding category. Next, the
machine is trained by supplying it with data representative of each category, for
which the machine then produces an output in the form of a vector of scores for every
category. The goal is to train the machine so that the vector of scores for a particular
piece of data has the highest possible score for the category to which it belongs. This
is achieved by computing a function that measures the error between the output
scores and a desired score pattern, after which the machine modifies its own
parameters, or weights, via a gradient vector—a function that adjusts weights
depending on whether the error would increase or decrease based on that adjustment—in order to reduce the error. Consequently, training can be an arduous process
for which the best results typically require incredibly extensive data dates and
hundreds of millions of weights [23].
More advanced machine learning algorithms attempt to completely emulate
neuronal design in software, simulating multi-layered neurons similar to the
perceptron in an architecture known as ‘deep learning’ [24]. Organized in a hierarchal tree, the output of one perceptron is fed forward to a connected perceptron to
compute complex tasks. The interaction between layers increases the difficulty in the
learning procedure due to the nonseparability of individual weights from the internal
recurrent response. Researchers have developed methods of mitigating the difficulties through feedback learning and error backpropagation schemes that predict
network behavior [24, 25].
Atomic Switch Networks for Neuroarchitectonics: Past, Present, Future
207
Modern neuromorphic system design strives to build the same adaptive learning
capacity found in biological neural systems into an electronic device. One way to
incorporate this system design is through the employment of machine learning
algorithms. Machine learning algorithms attempt to emulate neuronal design in
software. The learning process can be accomplished in a supervised or unsupervised
manner, with the key distinction being whether the data fed into the algorithm is
labeled (supervised) or unlabeled (unsupervised) [23]. Presently, supervised learning
is the most common form of machine learning, regardless of the algorithm
employed.
Historically, machine learning algorithms and techniques struggled to process
natural data sets in raw form and only through extensive work by engineers could
machine learning be tailored to perform tasks such as speech recognition [23]. This
has changed with the development of new methods, feature learning or representation learning, which are capable of identifying the minimum features that represents
each class of object for detection and/or classification within a data set. Those
methods led to the creation of the well-publicized machine learning technique
known as deep-learning that simulates a multi-layered neuron architecture. Its
purpose is for organizing unstructured data and applying this learned information
to various tasks, such as computer vision, speech recognition and bioinformatics. In
the case of supervised learning, the process starts by amassing a large data set where
each element of the data set is labeled with its corresponding category. Next, the
machine is trained by supplying it with data representative of each category, for
which the machine then produces an output in the form of a vector of scores for every
category. The goal is to train the machine so that the vector of scores for a particular
piece of data has the highest possible score for the category to which it belongs. This
is achieved by computing a function that measures the error between the output
scores and a desired score pattern, after which the machine modifies its own
parameters, or weights, via a gradient vector—a function that adjusts weights
depending on whether the error would increase or decrease based on that adjustment—in order to reduce the error. Consequently, training can be an arduous process
for which the best results typically require incredibly extensive data dates and
hundreds of millions of weights [23].
More advanced machine learning algorithms attempt to completely emulate
neuronal design in software, simulating multi-layered neurons similar to the
perceptron in an architecture known as ‘deep learning’ [24]. Organized in a hierarchal tree, the output of one perceptron is fed forward to a connected perceptron to
compute complex tasks. The interaction between layers increases the difficulty in the
learning procedure due to the nonseparability of individual weights from the internal
recurrent response. Researchers have developed methods of mitigating the difficulties through feedback learning and error backpropagation schemes that predict
network behavior [24, 25].
Atomic Switch Networks for Neuroarchitectonics: Past, Present, Future
207