Maximum Likelihood Estimation Notes | EduRev

: Maximum Likelihood Estimation Notes | EduRev

 Page 1


1
Maximum Likelihood Estimation
Econ 424/Amath 540
Eric Zivot
© Eric Zivot 2006
Eric Zivot
Summer 2012
Updated: July 26, 2012
Bernoulli Likelihood
likelihood.Bernoulli = function(theta, x) {
# theta   success probability parameter
# x       vector of data
n = length(x)
ans = theta^sum(x) * (1-theta)^(n-sum(x))
return(ans)
}
# plot Bernoulli likelihood
> x = rep(0,5)
> theta.vals = seq(0,1, length.out=10)
> like.vals = likelihood.Bernoulli(theta.vals, x)
> plot(theta.vals, like.vals, type="b", col="blue", lwd=2,
i "B lliLiklih df (00000)")
© Eric Zivot 2006
+      main="Bernoulli Likelihood for x=(0,0,0,0,0)")
Page 2


1
Maximum Likelihood Estimation
Econ 424/Amath 540
Eric Zivot
© Eric Zivot 2006
Eric Zivot
Summer 2012
Updated: July 26, 2012
Bernoulli Likelihood
likelihood.Bernoulli = function(theta, x) {
# theta   success probability parameter
# x       vector of data
n = length(x)
ans = theta^sum(x) * (1-theta)^(n-sum(x))
return(ans)
}
# plot Bernoulli likelihood
> x = rep(0,5)
> theta.vals = seq(0,1, length.out=10)
> like.vals = likelihood.Bernoulli(theta.vals, x)
> plot(theta.vals, like.vals, type="b", col="blue", lwd=2,
i "B lliLiklih df (00000)")
© Eric Zivot 2006
+      main="Bernoulli Likelihood for x=(0,0,0,0,0)")
2
Bernoulli Likelihood
Maximum of the likelihood function occurs 
at ? = 0
Note: for x=(0,0,0,0,0) the sample mean is 
0.
© Eric Zivot 2006
Bernoulli Likelihood
Maximum of the likelihood function occurs 
at ? = 1
Note: for x=(1,1,1,1,1) the sample mean is 
1.
© Eric Zivot 2006
Page 3


1
Maximum Likelihood Estimation
Econ 424/Amath 540
Eric Zivot
© Eric Zivot 2006
Eric Zivot
Summer 2012
Updated: July 26, 2012
Bernoulli Likelihood
likelihood.Bernoulli = function(theta, x) {
# theta   success probability parameter
# x       vector of data
n = length(x)
ans = theta^sum(x) * (1-theta)^(n-sum(x))
return(ans)
}
# plot Bernoulli likelihood
> x = rep(0,5)
> theta.vals = seq(0,1, length.out=10)
> like.vals = likelihood.Bernoulli(theta.vals, x)
> plot(theta.vals, like.vals, type="b", col="blue", lwd=2,
i "B lliLiklih df (00000)")
© Eric Zivot 2006
+      main="Bernoulli Likelihood for x=(0,0,0,0,0)")
2
Bernoulli Likelihood
Maximum of the likelihood function occurs 
at ? = 0
Note: for x=(0,0,0,0,0) the sample mean is 
0.
© Eric Zivot 2006
Bernoulli Likelihood
Maximum of the likelihood function occurs 
at ? = 1
Note: for x=(1,1,1,1,1) the sample mean is 
1.
© Eric Zivot 2006
3
Normal Likelihood
likelihood.normal.mu = function(mu, sig2=1, x) {
# mu      mean of normal distribution for given sig2
# x       vector of data
n = length(x)
a1 = (2*pi*sig2)^-(n/2)
a2 = -1/(2*sig2)
y = (x-mu)^2 y  (x mu) 2
ans = a1*exp(a2*sum(y))
return(ans) 
}
# generate N(0,1) data
> n = 50
> x = rnorm(n, mean=0, sd=1)
# compute normal likelihood as function of mu
© Eric Zivot 2006
> mu.vals = seq(-1,1, length.out=100)
> like.vals = rep(0,length(mu.vals))
> for (i in 1:length(like.vals)) {
+   like.vals[i] = likelihood.normal.mu(mu.vals[i], sig2=1, x=x)
+ }
Normal Likelihood for µ
Maximum of the 
likelihood does not likelihood does not 
occur at µ = 0. The 
maximum is at 
1
1
ˆ 0.0344
n
i
i
x
n
?
?
??
?
© Eric Zivot 2006
Page 4


1
Maximum Likelihood Estimation
Econ 424/Amath 540
Eric Zivot
© Eric Zivot 2006
Eric Zivot
Summer 2012
Updated: July 26, 2012
Bernoulli Likelihood
likelihood.Bernoulli = function(theta, x) {
# theta   success probability parameter
# x       vector of data
n = length(x)
ans = theta^sum(x) * (1-theta)^(n-sum(x))
return(ans)
}
# plot Bernoulli likelihood
> x = rep(0,5)
> theta.vals = seq(0,1, length.out=10)
> like.vals = likelihood.Bernoulli(theta.vals, x)
> plot(theta.vals, like.vals, type="b", col="blue", lwd=2,
i "B lliLiklih df (00000)")
© Eric Zivot 2006
+      main="Bernoulli Likelihood for x=(0,0,0,0,0)")
2
Bernoulli Likelihood
Maximum of the likelihood function occurs 
at ? = 0
Note: for x=(0,0,0,0,0) the sample mean is 
0.
© Eric Zivot 2006
Bernoulli Likelihood
Maximum of the likelihood function occurs 
at ? = 1
Note: for x=(1,1,1,1,1) the sample mean is 
1.
© Eric Zivot 2006
3
Normal Likelihood
likelihood.normal.mu = function(mu, sig2=1, x) {
# mu      mean of normal distribution for given sig2
# x       vector of data
n = length(x)
a1 = (2*pi*sig2)^-(n/2)
a2 = -1/(2*sig2)
y = (x-mu)^2 y  (x mu) 2
ans = a1*exp(a2*sum(y))
return(ans) 
}
# generate N(0,1) data
> n = 50
> x = rnorm(n, mean=0, sd=1)
# compute normal likelihood as function of mu
© Eric Zivot 2006
> mu.vals = seq(-1,1, length.out=100)
> like.vals = rep(0,length(mu.vals))
> for (i in 1:length(like.vals)) {
+   like.vals[i] = likelihood.normal.mu(mu.vals[i], sig2=1, x=x)
+ }
Normal Likelihood for µ
Maximum of the 
likelihood does not likelihood does not 
occur at µ = 0. The 
maximum is at 
1
1
ˆ 0.0344
n
i
i
x
n
?
?
??
?
© Eric Zivot 2006
4
Bernoulli Log-Likelihood
© Eric Zivot 2006
Normal Log-Likelihood
Maximum of the log-
likelihood does not occur 
at µ = 0. The maximum is 
at 
1
1
ˆ 0.0344
n
i
i
x
n
?
?
??
?
© Eric Zivot 2006
Page 5


1
Maximum Likelihood Estimation
Econ 424/Amath 540
Eric Zivot
© Eric Zivot 2006
Eric Zivot
Summer 2012
Updated: July 26, 2012
Bernoulli Likelihood
likelihood.Bernoulli = function(theta, x) {
# theta   success probability parameter
# x       vector of data
n = length(x)
ans = theta^sum(x) * (1-theta)^(n-sum(x))
return(ans)
}
# plot Bernoulli likelihood
> x = rep(0,5)
> theta.vals = seq(0,1, length.out=10)
> like.vals = likelihood.Bernoulli(theta.vals, x)
> plot(theta.vals, like.vals, type="b", col="blue", lwd=2,
i "B lliLiklih df (00000)")
© Eric Zivot 2006
+      main="Bernoulli Likelihood for x=(0,0,0,0,0)")
2
Bernoulli Likelihood
Maximum of the likelihood function occurs 
at ? = 0
Note: for x=(0,0,0,0,0) the sample mean is 
0.
© Eric Zivot 2006
Bernoulli Likelihood
Maximum of the likelihood function occurs 
at ? = 1
Note: for x=(1,1,1,1,1) the sample mean is 
1.
© Eric Zivot 2006
3
Normal Likelihood
likelihood.normal.mu = function(mu, sig2=1, x) {
# mu      mean of normal distribution for given sig2
# x       vector of data
n = length(x)
a1 = (2*pi*sig2)^-(n/2)
a2 = -1/(2*sig2)
y = (x-mu)^2 y  (x mu) 2
ans = a1*exp(a2*sum(y))
return(ans) 
}
# generate N(0,1) data
> n = 50
> x = rnorm(n, mean=0, sd=1)
# compute normal likelihood as function of mu
© Eric Zivot 2006
> mu.vals = seq(-1,1, length.out=100)
> like.vals = rep(0,length(mu.vals))
> for (i in 1:length(like.vals)) {
+   like.vals[i] = likelihood.normal.mu(mu.vals[i], sig2=1, x=x)
+ }
Normal Likelihood for µ
Maximum of the 
likelihood does not likelihood does not 
occur at µ = 0. The 
maximum is at 
1
1
ˆ 0.0344
n
i
i
x
n
?
?
??
?
© Eric Zivot 2006
4
Bernoulli Log-Likelihood
© Eric Zivot 2006
Normal Log-Likelihood
Maximum of the log-
likelihood does not occur 
at µ = 0. The maximum is 
at 
1
1
ˆ 0.0344
n
i
i
x
n
?
?
??
?
© Eric Zivot 2006
5
R optimize() function
# use optimize() to maximize or minimize function of one variable
test.fun = function(x) {
return(x^2)
}
> ans = optimize(test.fun, lower=-1, upper=1, maximum=FALSE)
> class(ans)
[1] "list"
> names(ans)
[1] "minimum"   "objective"
>
.6 0.8 1.0
f(x)=x^2
© Eric Zivot 2006
> ans
$minimum
[1] -2.776e-17
$objective
[1] 7.704e-34
-1.0 -0.5 0.0 0.5 1.0
0.0 0.2 0.4 0
x
f(x)
R optim() function
# use optim() to minimize functions of multiple variables
test.fun = function(theta) {
ans = theta[1]^2 + theta[2]^2
return(ans)
} }
# set starting values for optimizer
> theta.start = c(1,1)
# optimize function
> ans = optim(par=theta.start, fn=test.fun,
+             method="BFGS")
> class(ans)
[1] "list"
© Eric Zivot 2006
[]
> names(ans)
[1] "par"         "value"       "counts"      "convergence"
[5] "message" 
Read More
Offer running on EduRev: Apply code STAYHOME200 to get INR 200 off on our premium plan EduRev Infinity!