139
Clustering and Classification
Once again and due to the multiplicative nature Equation 7.10 that defines the
likelihood function, instead of optimizing the likelihood function p(D|θ) directly,
we often use the log of this function. Using Equation 7.10, we obtain a new function
l(θ) defined as follows:
∑
n
l( ) =
ln p X k |q
q
(
)
(7.13)
k =1
This function is often referred to as the log likelihood function. Next, in order to
obtain the best estimation, θ ˆ , we need to see how the log likelihood function can be
maximized. A simple approach is to utilize the gradient of this log likelihood function to find θ ˆ that maximizes l(θ). Calculating the gradient of the likelihood function
in terms of θ, we have
∇ =
l
∇ ln p X k |q
q
q
(
)
(7.14)
∑
n
k =1
Next, all we have to do is to set ∇ θ l equal to zero to find θ ˆ , i.e., our best estimation of
parameters would be the value θ that satisfies the equation ∇ θ l = 0. The exact form
of this equation evidently depends on the exact form of the probability functions
p(X k |θ)’s. For many practical choices of p(X k |θ), the equation ∇ θ l = 0 becomes too
complicated to be solved manually, and, therefore, computational software such as
MATLAB would be needed to find the maximum likelihood estimation θ ˆ .
Example 7.5
In this example, we show how to perform MLE in MATLAB. In MATLAB, one
can utilize the command “mle” to estimate the parameters assuming a certain
probability distribution. As can be seen in the following code, we first produce a
vector of 400 normally distributed random samples. In the command “mle”, we
need to determine the form of the distribution under which the data are produced
and what “mle” estimates are the set of parameters for the chosen distribution
form. There are some other commands in MATLAB that apply “mle” estimator
to estimate parameters of some popular distributions such as binomial, exponential, gamma, etc. The following code shows the command “binofit” that uses
“mle” estimator to estimate parameters of binomial distribution and “expfit”
that estimates parameters of the exponential distribution.
x1=randn(1,400);
mle(normal,x1);
r=binornd(100,.9);
[phat,pci]=binofit(r,100);
lifetimes= exprnd(700,100,1);
[muhat,muci]= expfit(lifetimes);
It is important to note that, in practice, the type of distribution function producing
the data is unknown. In the previous example, even though we have produced
Clustering and Classification
Once again and due to the multiplicative nature Equation 7.10 that defines the
likelihood function, instead of optimizing the likelihood function p(D|θ) directly,
we often use the log of this function. Using Equation 7.10, we obtain a new function
l(θ) defined as follows:
∑
n
l( ) =
ln p X k |q
q
(
)
(7.13)
k =1
This function is often referred to as the log likelihood function. Next, in order to
obtain the best estimation, θ ˆ , we need to see how the log likelihood function can be
maximized. A simple approach is to utilize the gradient of this log likelihood function to find θ ˆ that maximizes l(θ). Calculating the gradient of the likelihood function
in terms of θ, we have
∇ =
l
∇ ln p X k |q
q
q
(
)
(7.14)
∑
n
k =1
Next, all we have to do is to set ∇ θ l equal to zero to find θ ˆ , i.e., our best estimation of
parameters would be the value θ that satisfies the equation ∇ θ l = 0. The exact form
of this equation evidently depends on the exact form of the probability functions
p(X k |θ)’s. For many practical choices of p(X k |θ), the equation ∇ θ l = 0 becomes too
complicated to be solved manually, and, therefore, computational software such as
MATLAB would be needed to find the maximum likelihood estimation θ ˆ .
Example 7.5
In this example, we show how to perform MLE in MATLAB. In MATLAB, one
can utilize the command “mle” to estimate the parameters assuming a certain
probability distribution. As can be seen in the following code, we first produce a
vector of 400 normally distributed random samples. In the command “mle”, we
need to determine the form of the distribution under which the data are produced
and what “mle” estimates are the set of parameters for the chosen distribution
form. There are some other commands in MATLAB that apply “mle” estimator
to estimate parameters of some popular distributions such as binomial, exponential, gamma, etc. The following code shows the command “binofit” that uses
“mle” estimator to estimate parameters of binomial distribution and “expfit”
that estimates parameters of the exponential distribution.
x1=randn(1,400);
mle(normal,x1);
r=binornd(100,.9);
[phat,pci]=binofit(r,100);
lifetimes= exprnd(700,100,1);
[muhat,muci]= expfit(lifetimes);
It is important to note that, in practice, the type of distribution function producing
the data is unknown. In the previous example, even though we have produced