Test: Mutual Information - Electronics and Communication Engineering (ECE) MCQ

Mutual Information measures the amount of information that one Random variable contains about another Random variable.
The mutual information between two jointly distributed discrete Random variables X and Y is given by:

In terms of Entropy, this is written as:
I(X ; Y) = H(X) – H(X/Y)        ---(1)
(Option (b) is correct)
Also, the conditional entropy states that:
H(X, Y) = H(Y/X) + H(X)
H(X, Y) = H(X/Y) + H(Y)
From above equations we can write:
H(X/Y) = H(X, Y) – H(Y)        ---(2)
Using Equations (1) and (2), we can write:
I(X ; Y) = H(X) – [H(X, Y) – H(Y)]
I(X ; Y) = H(X) + H(Y) – H(X, Y)        ---(3)
(Option (c) is correct)

Statement (A):- A given source will have maximum entropy if messages produced are equally probable. So statement A given is wrong.
Statement (B):- From Shannon's channel capacity theorem:-

Statement B is wrong.
Statement (c):-
Band rate = 
For binary transmission M = 2
Band rate  
Band rate = Bitrate
Statement (C) is correct.
Statement (D):- it is False
Mutual information I (X, Y) = H (X) – H (X/Y)
= H (Y) – H (Y/X)
For binary symmetric channel
I (X, Y) is dependent on source
Probabilities it is not constant.
Statement (E):- It is correct.
Nat, bit, or Hartley is unit of Information,
Option (3) is correct.

Mutual information of two random variables is a measure to tell how much one random variable tells about the other.
It is mathematically defined as:
I(X₁, X₂) = H(X₁) – H(X₁/X₂)
Application:
Since X₁ and X₂ are independent, we can write:
H(X₁/X₂) = H(X₁)
I(X₁,X₂ ) = H(X₁) – H(X₁)
= 0

If r₀ is received:
P(m₀) P(r_0­/m₀) = 0.6 × 0.6 = 0.36
Also,
P(m₁) P(r0­/m₁) = 0.4 × 0 = 0
If r₁ is received:
P(m₀) P (r₁/m₀) =  (0.6) (0.3)
= 0.18
P(m₁­) P(r₁/m₁)
= (0.4) (0.7)
= 0.28
If r₂ is received:
P(m₀) P(r₂/m₀)
(0.6) (0.1)
= 0.06
P(m₁) P(r₂/m₁)
= (0.4) (0.3)
= 0.12
Probability of correct detection
P_c = 0.12 + 0.28 + 0.36
= 0.76
P_e = 1 – 0.76
= 0.24

If r₀ is received
P (m₀) P (r₀ / m₀) = 0.5 × 0.5 = 0.30
P (m₁) P (r₀ / m₁) = 0.5 × 0 = 0
If r_1­ is received
P (m₁) P(r₁/m₀)
(0.5) (0.3)
= 0.15
P (m₁) P(r₁/m₁)
(0.5) (0.7)
= 0.35
If r₂ is received
P (m₀) P(r₂/m₀)
= (0.5) (0.1)
= 0.05
P (m₁) P(r₂/m₁)
(0.5) (0.3)
= 0.15
Probability of correct = 0.15 + 0.35 + 0.30
⇒ 0.80
Probability of error = 1 – 0.80
= 0.20 

P(y₁) = 3/8
P(y₂) = 5/8
Mutual Information I(xy)
I(xy) = H(x) - H(x/y)
= H(y) - H(y/x)
H(y) - Σ P(y_j) log2 P(y_i)
= - [0.375 log₂(0.375) + 0.625 log₂(0.625)]

I(xy) = 0.05 bits/symbol

I (xy) = (1 - p) H (x)

= 0.4 [2 log 2] ( log used is base 2)
⇒ 0.8

The channel matrix can be represented as