independent objective validation criteria are needed to ensure that segmentations
are not subject to bias.
Another possibility to improve performance and throughput is the use of
machine learning. Machine learning has been used with some success in
single-particle electron cryo-microscopy data processing in the context of particle
picking [105–109] and has also been used for segmentation tests on tomograms of
silicon pillars obtained by scanning transmission electron microscopy [110].
Applications to biological (cryo-)tomograms are lacking. One reason is that conventional machine-learning approaches do not process data in their raw form but
require careful engineering and considerable expertise to design feature extractors
that convert raw voxel values into an internal representation suitable as input for
learning algorithms [111]. However, a new class of machine learning approaches
termed deep learning have recently been introduced that simplify this task [112].
The underlying learning methods are fed with raw data and automatically discover
the representations needed for detection or classification. The deep learning
approach uses multiple levels of representations, each at a slightly more abstract
level than the previous one. The layers of representations are not designed by
human operators but are learned through the general purpose representation
learning algorithm. These types of deep learning approaches were shown to be
spectacularly successful in some application domains. In particular, deep convolutional neural networks have done exceedingly well in image recognition tasks.
Before 2012 no computer algorithm was able to surpass the 25% mark in error rates
for the ImageNet competition, which concerns classifying and locating different
objects in a large set of natural images [113]. With the introduction of deep convolutional neural networks in 2012, this rate dropped to 16% [114] and, after
additional improvements, is now reduced to a few percent [115–117]. Applications
of deep learning approaches are already appearing for particle picking tasks in
single-particle electron cryo-microscopy [118–120] and neuron tracing in block
face scanning electron microscopy reconstructions [121] but have not penetrated the
field of electron (cryo-)tomography yet. One downside of deep convolutional
networks is the need for a sufficient amount of ‘ground-truth’ training data that are
hard to obtain, especially for electron cryo-tomograms. Another issue is that poorly
defined training and test data can lead to significant bias [122]. Deep reinforcement
learning approaches, which might remedy both issues, are under development and
show some promise [123–125] but are not well enough matured yet to be useful in
the electron (cryo-)tomography context. Perhaps a combination of supervised deep
learning and carefully designed crowdsourcing will present the next step forward.
Acknowledgements Writing of this article was made possible by support from the National
Institutes of Health (grant numbers P01-GM121203 and R01-GM119948).
312
N. Volkmann
are not subject to bias.
Another possibility to improve performance and throughput is the use of
machine learning. Machine learning has been used with some success in
single-particle electron cryo-microscopy data processing in the context of particle
picking [105–109] and has also been used for segmentation tests on tomograms of
silicon pillars obtained by scanning transmission electron microscopy [110].
Applications to biological (cryo-)tomograms are lacking. One reason is that conventional machine-learning approaches do not process data in their raw form but
require careful engineering and considerable expertise to design feature extractors
that convert raw voxel values into an internal representation suitable as input for
learning algorithms [111]. However, a new class of machine learning approaches
termed deep learning have recently been introduced that simplify this task [112].
The underlying learning methods are fed with raw data and automatically discover
the representations needed for detection or classification. The deep learning
approach uses multiple levels of representations, each at a slightly more abstract
level than the previous one. The layers of representations are not designed by
human operators but are learned through the general purpose representation
learning algorithm. These types of deep learning approaches were shown to be
spectacularly successful in some application domains. In particular, deep convolutional neural networks have done exceedingly well in image recognition tasks.
Before 2012 no computer algorithm was able to surpass the 25% mark in error rates
for the ImageNet competition, which concerns classifying and locating different
objects in a large set of natural images [113]. With the introduction of deep convolutional neural networks in 2012, this rate dropped to 16% [114] and, after
additional improvements, is now reduced to a few percent [115–117]. Applications
of deep learning approaches are already appearing for particle picking tasks in
single-particle electron cryo-microscopy [118–120] and neuron tracing in block
face scanning electron microscopy reconstructions [121] but have not penetrated the
field of electron (cryo-)tomography yet. One downside of deep convolutional
networks is the need for a sufficient amount of ‘ground-truth’ training data that are
hard to obtain, especially for electron cryo-tomograms. Another issue is that poorly
defined training and test data can lead to significant bias [122]. Deep reinforcement
learning approaches, which might remedy both issues, are under development and
show some promise [123–125] but are not well enough matured yet to be useful in
the electron (cryo-)tomography context. Perhaps a combination of supervised deep
learning and carefully designed crowdsourcing will present the next step forward.
Acknowledgements Writing of this article was made possible by support from the National
Institutes of Health (grant numbers P01-GM121203 and R01-GM119948).
312
N. Volkmann