Chapter 4
Advanced Neural Networks
Abstract In this chapter, we explain the structure of the two types of neural
networks that have been the mainstays of deep learning in recent years, following
the words of physics in the previous chapter. A convolutional neural network has a
structure that emphasizes the spatial proximity in input data. Also, recurrent neural
networks have a structure to learn input data in time series. You will learn how to
provide a network structure that respects the characteristics of data.
In Chap. 3, using physics language, we have described the introduction of the most
basic neural network (forward propagation neural network) and the associated error
function and the gradient calculation of the empirical error (error backpropagation
method). This chapter is for the following two representative examples of a recent
deep learning architecture:
• Convolutional neural network
• Recurrent neural network
and we will explain the structure of these and related topics.
4.1 Convolutional Neural Network
4.1.1 Convolution
Consider a two-dimensional image. For a grayscale image, the input value is x ij as
the ij th pixel value, and for a color image, it is x
c
ij as the ij th pixel value, with c for
the cth color channel. So we naturally consider the tensor structure. As an example,
let us consider d I J as
• Probability that a cat is shown around i ≈ I, j ≈ J .
(4.1)
© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021
A. Tanaka et al., Deep Learning and Physics, Mathematical Physics Studies,
https://doi.org/10.1007/978-981-33-6108-9_4
57
Advanced Neural Networks
Abstract In this chapter, we explain the structure of the two types of neural
networks that have been the mainstays of deep learning in recent years, following
the words of physics in the previous chapter. A convolutional neural network has a
structure that emphasizes the spatial proximity in input data. Also, recurrent neural
networks have a structure to learn input data in time series. You will learn how to
provide a network structure that respects the characteristics of data.
In Chap. 3, using physics language, we have described the introduction of the most
basic neural network (forward propagation neural network) and the associated error
function and the gradient calculation of the empirical error (error backpropagation
method). This chapter is for the following two representative examples of a recent
deep learning architecture:
• Convolutional neural network
• Recurrent neural network
and we will explain the structure of these and related topics.
4.1 Convolutional Neural Network
4.1.1 Convolution
Consider a two-dimensional image. For a grayscale image, the input value is x ij as
the ij th pixel value, and for a color image, it is x
c
ij as the ij th pixel value, with c for
the cth color channel. So we naturally consider the tensor structure. As an example,
let us consider d I J as
• Probability that a cat is shown around i ≈ I, j ≈ J .
(4.1)
© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021
A. Tanaka et al., Deep Learning and Physics, Mathematical Physics Studies,
https://doi.org/10.1007/978-981-33-6108-9_4
57