5.2 Various Sampling Methods
85
next chapter. Even in such a situation, there are cases in which sampling is possible. 7
As long as sampling is possible, the expectation value can be calculated by a method
that replaces the expectation value with the sample average value, based on the law
of large numbers (5.14) and the central limit theorem (5.16).
Thus, sampling techniques are an important concept to understand in machine
learning methods. In the following, assuming that random numbers (sampling
from a uniform probability) are given, 8 we explain some methods for performing
sampling from more complex probabilities.
5.2.1 Inverse Transform Sampling
Assuming that it is possible to sample z which follows a uniform distribution, how
can we sample x following a probability distribution P (x)? First, the probability of
finding x is
P (x)dx .
(5.23)
Suppose that this is written as a total derivative of certain function F (x) as follows:
dF (x) = P (x)dx.
(5.24)
Since the probability of finding z following a uniform distribution is proportional to
dz, (5.24) means that, with regarding F (x) = z and using the inverse function F −1
of F ,
x := F
−1 (z) ,
(5.25)
7 Although not described in this book, the calculation of a probability distribution called a posterior
distribution in Bayesian estimation corresponds to this case.
8 As a matter of fact, it is impossible to make a true random number with a classical calculator.
A definition of random numbers can be given by Kolmogorov complexity, so it is not possible to
provide a random sequence of infinite length. In many cases in physics, a reproducible sequence
of random numbers is required for practical use, and so, pseudo-random numbers, which are
sequences of finite period numbers that can be reproduced and are statistically unbiased, are used.
Von Neumann said: “Any one who considers arithmetical methods of producing random digits is, of
course, in a state of sin.” But it is used in modern times with a lot of ingenuity. It is also known that a
statistically biased pseudo-random number generation method can lead to incorrect calculations of
the transition temperature of the simulated Ising model [60]. Not only that it should be statistically
unbiased, but also a long periodicity where the same sequence does not appear is also required
for large-scale simulations. More recently, the one published by Makoto Matsumoto and Takuji
Nishimura, called Mersenne Twister, has been used worldwide [61]. It has advantages such as long
periods, uniform distribution, and fast generation. Other pseudo-random number generators, such
as XorShift and MIXMAX, have also been proposed [62]. We suggest that readers refer to other
references for more information [63].
85
next chapter. Even in such a situation, there are cases in which sampling is possible. 7
As long as sampling is possible, the expectation value can be calculated by a method
that replaces the expectation value with the sample average value, based on the law
of large numbers (5.14) and the central limit theorem (5.16).
Thus, sampling techniques are an important concept to understand in machine
learning methods. In the following, assuming that random numbers (sampling
from a uniform probability) are given, 8 we explain some methods for performing
sampling from more complex probabilities.
5.2.1 Inverse Transform Sampling
Assuming that it is possible to sample z which follows a uniform distribution, how
can we sample x following a probability distribution P (x)? First, the probability of
finding x is
P (x)dx .
(5.23)
Suppose that this is written as a total derivative of certain function F (x) as follows:
dF (x) = P (x)dx.
(5.24)
Since the probability of finding z following a uniform distribution is proportional to
dz, (5.24) means that, with regarding F (x) = z and using the inverse function F −1
of F ,
x := F
−1 (z) ,
(5.25)
7 Although not described in this book, the calculation of a probability distribution called a posterior
distribution in Bayesian estimation corresponds to this case.
8 As a matter of fact, it is impossible to make a true random number with a classical calculator.
A definition of random numbers can be given by Kolmogorov complexity, so it is not possible to
provide a random sequence of infinite length. In many cases in physics, a reproducible sequence
of random numbers is required for practical use, and so, pseudo-random numbers, which are
sequences of finite period numbers that can be reproduced and are statistically unbiased, are used.
Von Neumann said: “Any one who considers arithmetical methods of producing random digits is, of
course, in a state of sin.” But it is used in modern times with a lot of ingenuity. It is also known that a
statistically biased pseudo-random number generation method can lead to incorrect calculations of
the transition temperature of the simulated Ising model [60]. Not only that it should be statistically
unbiased, but also a long periodicity where the same sequence does not appear is also required
for large-scale simulations. More recently, the one published by Makoto Matsumoto and Takuji
Nishimura, called Mersenne Twister, has been used worldwide [61]. It has advantages such as long
periods, uniform distribution, and fast generation. Other pseudo-random number generators, such
as XorShift and MIXMAX, have also been proposed [62]. We suggest that readers refer to other
references for more information [63].