Chapter 5
Sampling
Abstract In the situation where the training is performed, it is assumed that the
input data is given by a probability distribution. It is often necessary to calculate the
expectation value of the function of various input values given by the probability
distribution. In this chapter, we will look at the method and the necessity of
“sampling,” which is the method of performing the calculation of the expectation
value. The frequently used concepts in statistical mechanics, such as the law of
large numbers, the central limit theorem, the Markov chain Monte Carlo method,
the principle of detailed balance, the Metropolis method, and the heat bath method,
are also used in machine learning. Familiarity with common concepts in physics and
machine learning can lead to an understanding of both.
As we explained in Chap. 2, in this book we define machine learning as:
1. Assuming that there exists a probability distribution P (x, d) that generates data
2. Adjust parameter J of probability distribution Q J (x, d) to approach P (x, d)
In order to introduce neural networks in Chaps. 3 and 4, we applied our knowledge
of statistical mechanics to this point of view. The output of the deep neural network
can be considered as the expectation value of d for the conditional probability
distribution Q J (d|x) given the input x. However, one may often want to perform
a sampling according to Q J (d|x) as the probability of occurrence for d, instead of
taking the expectation value. For example, let us consider
Q J (d|x) = Probability of classifying what is in the input image x as d.
Suppose
x = Photo of grandfather laughing with his dog ,
© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021
A. Tanaka et al., Deep Learning and Physics, Mathematical Physics Studies,
https://doi.org/10.1007/978-981-33-6108-9_5
77
Sampling
Abstract In the situation where the training is performed, it is assumed that the
input data is given by a probability distribution. It is often necessary to calculate the
expectation value of the function of various input values given by the probability
distribution. In this chapter, we will look at the method and the necessity of
“sampling,” which is the method of performing the calculation of the expectation
value. The frequently used concepts in statistical mechanics, such as the law of
large numbers, the central limit theorem, the Markov chain Monte Carlo method,
the principle of detailed balance, the Metropolis method, and the heat bath method,
are also used in machine learning. Familiarity with common concepts in physics and
machine learning can lead to an understanding of both.
As we explained in Chap. 2, in this book we define machine learning as:
1. Assuming that there exists a probability distribution P (x, d) that generates data
2. Adjust parameter J of probability distribution Q J (x, d) to approach P (x, d)
In order to introduce neural networks in Chaps. 3 and 4, we applied our knowledge
of statistical mechanics to this point of view. The output of the deep neural network
can be considered as the expectation value of d for the conditional probability
distribution Q J (d|x) given the input x. However, one may often want to perform
a sampling according to Q J (d|x) as the probability of occurrence for d, instead of
taking the expectation value. For example, let us consider
Q J (d|x) = Probability of classifying what is in the input image x as d.
Suppose
x = Photo of grandfather laughing with his dog ,
© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021
A. Tanaka et al., Deep Learning and Physics, Mathematical Physics Studies,
https://doi.org/10.1007/978-981-33-6108-9_5
77