Modes of convergence - Probability and probability Distributions, CSIR-NET Mathematical Sciences | Mathematics for IIT JAM, GATE, CSIR NET, UGC NET PDF Download

Various Modes of Convergence

Definitions 

• (convergence in probability) A sequence of random variables {X_n} is said to converge in probability to a random variable X as n →∞if for any ε>0 we have 

We write  or plim X_n = X.

• (convergence in distribution) Let F and F_n be the distribution functions of X and X_n, respectively. The sequence of random variables {X_n} is said to converge in distribution to a random variable X as n →∞if 

for all z ∈ R and z is a continuity points of F. We write 

•  (almost sure convergence) We say that a sequence of random variables {X_n} converges almost surely or with probability 1 to a random variable X as n →∞if 

We write 

•  (L^r convergence) A sequence of random variables {X_n} is said to converge in L^r norm to a random variable X as n →∞if for some r>0

We denote a   If r =2 , it is called mean square convergence and denoted as 

Relationship among various modes of convergence 

Example 1 Convergence in distribution does not imply convergence in probability.

⇒ Let Ω ={ω₁,ω₂,ω₃,ω₄}. Define the random variables X_n and X such that 

Moreover, we assign equal probability to each event. Then,

Since F_n (x)=F (x) for all n, it is trivial that  However,

Note that |X_n (ω)−X (ω)|= 1 for all n and ω. Hence, .

Example 2 Convergence in probability does not imply almost sure convergence.

⇒ Consider the sequence of independent random variables {X_n} such that 

Obviously for any 0 <ε<1, we have

Hence  In order to show   , we need the following lemma.
 

Proof. Let C ={ω : X_n (ω)→ X (ω) asn →∞},A(ε)={ω : ω ∈ A_n (ε) i.o.}. Then, P (C) = 1 if and only if P (A(ε)) = 0 for all ε>0. However, B_m (ε) is a decreasing sequence of events, B_m (ε)↓ A(ε) asm →∞and so P (A(ε)) = 0 if and only if P (B_m (ε))→∞as m →∞ . Continuing the counter-example, we have 

Hence,  

Example 4 Convergence in probability does not imply convergence in L^r −norm.

⇒ Let {X_n} be a random variable such that 

Then, for any ε>0 we have

Hence, . However, for each r>0, 

Hence, 

Some useful theorems

Theorem 5 Let {X_n} be a random vector with a fixed finite number of elements. Let g be a real-valued function continuous at a constant vector point α. Then

⇒By continuity of g at α, forany ε >0 we can find δ suchthat    ||X_n −α||    <δ implies |g(X_n) − g(α)| <ε . Therefore,  

Theorem 6 Suppose that    where α is non-stochastic. Then 

• Note the condition thatwhere α is non-stochastic. If α is also a random vector,  A counter-example is given by

Then,

Theorem 7 Let {X_n} be a random vector with a fixed finite number of elements. Let g be a continuous real-valued function . Then 

Theorem 8 Suppose 

Inequalities frequently used in large sample theory 

Proposition 9 (Chebychev’s inequality) For ε>0

Proposition 10 (Markov’s inequality) For ε>0 and p>0 

Proposition 11 (Jensen’s inequality) If a function φ is convex on an interval I containing the support of a random variable X, then 

φ(E(X)) ≤E (φ(X))

Proposition 12 (Cauchy-Schwartz inequality) For random variables X and Y 

Proposition 13 (H¨older’s inequality ) For any p≥1 

where 

Proposition 14 (Lianpunov’s inequality) If r>p>0, 

Proposition 15 (Minkowski’s inequality) For r ≥1, 

Proposition 16 (Lo`eve’s cr inequality) For r>0, 

where c_r =1when 0 <r≤1, and c_r = m^r−1when r>1.

Laws of Large Numbers

•  Suppose we have a set of observation X₁,X₂,···,X_n. A law of large numbers basically gives us the behavior of sample mean  when the number of observations n goes to infinity. It is needless to say that we need some restrictions(assumptions) on the behavior of each individual random variable X_i and on the relationship among  is. There are many versions of law of large numbers depending on what kind of restriction we are wiling to impose. The most generic version can be stated as

Given restrictions on the dependence, heterogeniety, and moments of a sequence of random variables  converges in some mode to a parameter value.

When the convergence is in probability sense, we call it a weak law of large numbers. When in almost sure sense, it is called a strong law of large numbers.

•  We will have a kind of trade-off between dependence or heterogeneity and existence of higher moments. As we want to allow for more dependence and heterogeneity, we have to accept the existence of higher moment, in general. 

Theorem 17 (Komolgorov SLLN I) Let {X_i} be a sequence of independently and identically distributed random variables. Then 

Remark 18 The above theorem requires the existence of the first moment only. However, the restriction on dependence and heterogeneity is quite severe. The theorem requires i.i.d.(random sample), which is rarely the case in econometrics. Note that the theorem is stated in necessary and sufficient form. Since almost sure convergence always implies convergence in probability, the theorem can be stated as . Then it is aweak law of large numbers. 

Theorem 19 (Komolgorov SLLN II) Let {X_i} be a sequence of independently distributed random variables with finite variances 

Remark 20 Here we allow for the heterogeneity of distributions in exchange for existence of the second moment. Still, they have to be independent. Intuitive explanation for the summation condition is that we should not have variances grow too fast so that we have a shrinking variance for the sample mean. 

•  The existence of the second moment is too strict in some sense. The following theorem might be a theoretical purism. But we can obtain a SLLN with milder restriction on the moments.

Theorem 21 (Markov SLLN) Let {X_i} be a sequence of independently distributed random variables with finite means E (X_i)=µ_i < ∞. If for some δ>0, 

Remark 22 When δ =1 , the theorem collapses to Komolgorov SLLN II. Here we don’t need the existence of the second moment. All we need is the existence of the moment of order (1+ δ) where δ>0.

• We now want to allow some dependence among . This modification is especially important when we are dealing with time series data which has a lot of dependence structure in it. 

Theorem 23 (Ergodic theorem) Let {X_i} be a (weakly) stationary and ergodic sequence with E|X_i| < ∞. Then,    

Remark 24 By stationarity, we have E (X_i)=µ for all i. And ergodicity enables us to have, roughly speaking, an estimate of µ as a sample mean of . Both stationarity and ergodicity are restrictions on dependence structure - which sometimes seem quite severe for econometric data. 

•  Inorder to allowboth dependence and heterogeneityweneed morespecific structureon the dependence of the data series called strong mixing and uniform mixing. The LLN’s in case of mixing requires some technical discussion. Anyway, one of the most important SLLN’s in econometrics is McLeish’s. 

Theorem 25 (McLeish) Let {X_i} be a sequence with a uniform mixing of size  or a strong mixing of size with finite means E (X_i)=µ_i. If for some  then 

•   Another form of SLLN important in econometric application is SLLN for a martingale difference sequence. A stochastic process X_t is called a martingale difference sequence if 

E (X_t |F _t−1) = 0 for all t
where F_t−1 = σ(X_t−1,X _t−2,···) i.e., information up to time (t−1). Theorem 26 (Chow) Let {X_i} be a martingale difference sequence. If for some  then

Central Limit Theorems 

• All CLT’s are meant to derive the distribution of sample mean as n →∞when appropriately scaled. We have many versions of CLT depending on our assumptions on the data. The easiest and most frequently cited CLT is Theorem 27 (Lindeberg-Levy CLT) Let {X_i} be a sequence of independently and identically distributed random variables. If V a^r(X_i)=σ² < ∞, then 

Remark 28 The conclusion of the theorem can be also written asWe requiresthe existence of the second moment even if we have i.i.d. sample. (Compare this with LLN).

Theorem 29 (Lindeberg-Feller CLT) Let {X_i} be a sequence of independently distributed random variables with   and distribution function F_i (x). Then  

and

if and only if

Remark 30 The condition is called ”Lindeberg condition”. The condition controls the tail behavior of X_i so that we have a proper distribution for scaled sample mean. We do not need identical distribution here. The condition is difficult to verify in practice. A search for a sufficient condition for the Lindeberg condition leads to the following CLT. 

Remark 32 We can show that the moment restrictions in the theorem are enough to obtain the Lindeberg condition. 

Remark 34 The above theorem allows some dependence structure but retains homogeneity through stationarity and ergodicity. 

Remark 36 The above CLT is quite general in the sense that we can allow reasonable dependence and heterogeneity structures to be applied to econometric data. However, as shown in the statement of the theorem, it is impractical to check the conditions of the theorem in practice.

• Finally, we will have a CLT which can be applied to a martingale difference sequence.