Methods of estimation, properties of estimators, CSIR-NET Mathematical Sciences | Mathematics for IIT JAM, GATE, CSIR NET, UGC NET PDF Download

2.3.1 Method of Moments

The Method of Moments is a simple technique based on the idea that the sample moments are “natural” estimators of population moments.

The k-th population moment of a random variable Y is

and the k-th sample moment of a sample Y₁,...,Y_n is

If Y₁,...,Y_n are assumed to be independent and identically distributed then the Method of Moments estimators of the distribution parameters ϑ₁,...,ϑ_p are obtained by solving the set of p equations:

Under fairly general conditions, Method of Moments estimators are asymptotically normal and asymptotically unbiased. However, they are not, in general, efficient.

Example 2.17. Let  We will find the Method of Moments estimators of µ and σ².

We have     So, the Method of Moments estimators of μ and σ2 satisfy the equations

Thus, we obtain

Estimators obtained by the Method of Moments are not always unique.
Example 2.18. Let
Poisson(λ). We will find the Method of Moments estimator of λ. We know that for this distribution E(Y_i) = var(Y_i) = λ. Hence By comparing the first and second population and sample moments we get two different estimators of the same parameter, 

2.3.2 Method of Maximum Likelihood

This method was introduced by R.A.Fisher and it is the most common method of constructing estimators. We will illustrate the method by the following simple example.

Example 2.19. Assume that  Bernoulli(p), i = 1,2,3,4, with probability of success equal to p, where  p belongs to the parameter space of only three elements. We want to estimate parameter p based on observations of the random sample Y = (Y₁,Y₂,Y₃,Y₄)^T. 
The joint pmf is 

The different values of the joint pmf for all p ∈ Θ are given in the table below

We see that   is largest when p = 1/4 It can be interpreted that when the observed value of the random sample is (0,0,0,0)^T the most likely value of the parameter p is  . Then, this value can be considered as an estimate of p. Similarly, we can conclude that when the observed value of the random sample is, for example, (0,1,1,0)^T, then the most likely value of the parameter is . Altogether, we have 

  = 1/4 if we observe ail failures or just one success;
 = 1/2 if we observe two failures and two successes:
 = 3/4  if we observe three successes and one failure or four successes.

Note that, for each point (y₁,y₂,y₃,y₄)^T, the estimate  is the value of parameter p for which the joint mass function, treated as a function of p, attains maximum (or its largest value).

Here, we treat the joint pmf as a function of parameter p for a given y. Such a function is called the likelihood function and it is denoted by L(p|y). 
Now we introducea formal definition of theMaximum Likelihood Estimator (MLE).

Definition 2.11. The MLE(ϑ) is the statistic  whose value for a given y satisfies the condition L

where L(ϑ|y) is the likelihood function for ϑ.

Properties of MLE

The MLEs are invariant, that is 

MLEs are asymptotically normal and asymptptically unbiased. Also, they are efficient, that is 

In this case, for large n, var is approximately equal to the CRLB. Therefore, for large n, 

 approximately. This is called the asymptotic distribution of 

 Example 2.20. Suppose that Y₁,...,Y_n are independent Poisson(λ) random variables. Then the likelihood is

We need to find the value of λ which maximizes the likelihood. This value will also maximize  which is easier to work with. Now, we have 

The value of λ which maximizes ℓ(λ|y) is the solution of dℓ(/dλ = 0. Thus, solving the equation

yields the estimator  which is the same as the Method of Moments estimator. The second derivative is negative for all λ hence,  indeed maximizes the log-likelihood

Example 2.21. Suppose that Y₁,...,Y_n are independent N(µ,σ²) random variables. Then the likelihood is 

and so the log-likelihood is

Thus, we have

and

Setting these equations to zero, we obtain

so that is the maximum likelihood estimator of µ, and

so that  is the maximum likelihood estimator of σ², which are the same as the Method of Moments estimators. 

2.3.3 Method of Least Squares

If Y₁,...,Y_n are independent random variables, which have the same variance and higher-order moments, and, if each E(Y_i) is a linear function of ϑ₁,...,ϑ_p, then the Least Squares estimates of ϑ₁,...,ϑ_p are obtained by minimizing

The Least Squares estimator of ϑ_j has minimum variance amongst all linear unbiased estimators of ϑ_j and is known as the best linear unbiased estimator (BLUE). If the Yis have a normal distribution, then the Least Squares estimator of ϑ_j is the Maximum Likelihood estimator, has a normal distribution and is the MVUE.

Example 2.22. Suppose that Y₁,...,Y_n1 are independentN(µ₁,σ²) random variables and that Y_n1+1,...,Y_n are independent N(µ²,σ²) random variables. Find the least squares estimators of µ₁ and µ₂.
Since

it is a linear function of µ₁ and µ₂. The Least Squares estimators are obtained by minimizing

Now,

and 

where n₂ = n−n₁. So, we estimate the mean of each group in the population by the mean of the corresponding sample. 

Example2.23. Supposethat  independently for i = 1,2,...,n, where xi is some explanatory variable. This is called the simple linear regression model. Find the least squares estimators of β₀ and β₁.

Since E(Y_i) = β₀ + β₁x_i, it is a linear function of β₀ and β₁. So we can obtain the least squares estimates by minimizing

Now

and

Substituting the first equation into the second one, we have

Hence, we have the estimators

These are the Least Squares estimators of the regression coefficient  β₀ and β₁.