Let H(X) denote the entropy of a discrete random variable taking ? possible distinct real values. Which of the following statements is/are necessarily true?a)H (X) ≤ log2 K bitsb)H (X) ≤ H(2X)c)H (X) ≤ H(2X)d)H (X) ≤ H(X2)Correct answer is option 'A,B,C'. Can you explain this answer?

Question

Let H(X) denote the entropy of a discrete random variable  taking ? possible distinct real values. Which of the following statements is/are necessarily true?a)H (X) ≤ log2 K bitsb)H (X) ≤ H(2X)c)H (X) ≤ H(2X)d)H (X) ≤ H(X2)Correct answer is option 'A,B,C'. Can you explain this answer?

Answer

Solution:

Statement A: H(X) ≤ log2 K bits

This statement is necessarily true as the entropy of a discrete random variable taking K possible distinct real values cannot be greater than log2 K. This is because each value can be represented using log2 K bits and the entropy is the expected number of bits needed to represent the value. Therefore, H(X) ≤ log2 K bits.

Statement B: H(X) ≤ H(2X)

This statement is necessarily true as the entropy of a random variable cannot increase when the variable is multiplied by a constant. Therefore, H(X) ≤ H(2X).

Statement C: H(X) ≤ H(2X)

This statement is necessarily true as the entropy of a random variable cannot increase when the variable is multiplied by a constant. Therefore, H(X) ≤ H(2X).

Statement D: H(X) ≤ H(X2)

This statement is not necessarily true as the entropy of a random variable can increase when the variable is squared. For example, if X is a uniform random variable taking values {1, 2, 3}, then H(X) = log2 3 bits. However, if X2 is defined as the square of X, then the possible values are {1, 4, 9} and the entropy is H(X2) = log2 3 + 1 bit, which is greater than H(X).

Answer

Given

H (X) is entropy of a discrete random variable X taking K possible distinct real values.

Let variable X is taking values as x_i so set of possible values is {x₁, x₂ , x₃.......x_k } .

The entropy will be,

Case-1 : When all values are equiprobable i.e. P(x_i) = 1/k for each distinct ' K' values, then entropy will be given as,

H ( X ) = log K

When we choose base as 2,

H ( X ) = log₂ K bits

We know that, in case of equal probability the entropy will be highest,

So, H(X) ≤ log₂ K is always true for any base value.

Hence, option (A) is correct.

Case-2 : Assuming a new random variable Y = 2X which is mapped by random variable X as shown below,

So, mapping will be one to one as we have linear relation between X and Y.

The random variable X = {x₁, x₂ , x₃.......x_k } in the same way random variable Y

Y = {y₁ = 2x₁, y₂ = 2x₂ , .......... y_K = 2x_K }

Along with values the probability of occurring each value will also be mapped and hence x_i and corresponding y_i will have identical probability, i. e. P(x_i) = P(y_i)

We can say that P( x₁ ) = P(y₁), P(x₂) = P(y₂)..........P(x_K) = P(y_K)

Probability of random variable X and Y are same, because one to one mapping therefore, entropy of random variable X and Y are same. i. e. H (Y = 2X ) = H ( X )

So we will never have situation of H ( X ) < h='' (2x='' />

Hence, option (B) is correct

Case-3 : Assume new random variable Y = 2^X , This Y = 2^X will give one to one mapping because different values of X provide different values of Y so that probability of random variable Y is same as probabilities of random variable X so that entropy

H(Y = 2^X) = H(X)

So, H ( X ) < h='' />^X) never occur

So, option (C) is also correct.

Case-4 : Assuming new random variable Y =X² .

Possibility-01 : Suppose we have 3 positive values of random variable X is x₁ = 1 , x₂ = 2, x₃ = 3 and assuming corresponding probabilities of x₁, x₂, x₃ are

respectively

i.e. X = {1, 2, 3}

Probability,

Thus, random variable, Y =X²= {1, 4, 9}

Here we can say different values of X gives different values of Y, it shows one to one mapping is possible between X and Y.

So, probabilities of random variable Y is same as probability of random variable X.

It means,

Thus we can say that entropy of H(X) = H(Y= X² )

Possibility-02 : Suppose we have 3 negative values of random variable X is x₁ = -1 , x₂ = -2 , x₃ = -3 and assuming corresponding probabilities of x₁, x₂, x₃ are

respectively

i.e. X = {-1, - 2, - 3}

Probability,

Thus, random variable, Y =X²= {1, 4, 9}

Here we can say different values of X gives different values of Y, it shows one to one mapping is possible between X and Y.

So, probabilities of random variable Y is same as probability of random variable X,

Thus we can say that entropy of H(X) = H(Y= X²) .

Possibility-03 : Suppose we have both positive and negative values of random varible X is x₁ = -1 , x₂ = 1, x₃ = 2 and assuming corresponding probabilities of x₁ , x₂ and x₃ are 1/5, 1/2 and 3/10.

i.e. X = {-1, 1, 2}

Thus random variable, Y =X²= {1, 1, 4}

Here we see different values of X gives same value of Y, it show one to one mapping is not possible here.

So, random variable Y have only 2-values i.e. Y = [1, 4]

So, probability of P(y₁ = 1) is sum of probability of P(x₁ =-1) and P(x₂ = 1) so, P(y₁= 1) = 1/5 + 1/2 = 7/10

Probability of P(y₂ = 4) remains same as probability of P(x₃ = 2) .

i.e. P(y₂ = 4) = P(x₃ = 2) = 3/10

So entropy of H (Y) is

Entropy of H (X) is,

= 0.4643 + 0.5 + 0.521089

=1.4854

From Case-4, it is clear that H (X) = H (X)² is possible only when random variable X have all positive or negative values and X can take combination of positive and negative values then H (X) > H (X²) that why option (D) is incorrect.

Hence, the correct options are (A), (B) & (C).

Similar Electronics and Communication Engineering (ECE) Doubts

A discrete random variable X has possible values x i= i2 i =1, 2, 3, 4 which occur with probabilities 0.4, 0.25, 0.15, 0.1,. The mean value

Top Courses for Electronics and Communication Engineering (ECE)View all

GATE ECE (Electronics) Mock Test Series 2025

Network Theory (Electric Circuits)

General Aptitude for GATE

Crash Course: Electronic Communication Engineering (ECE)

Signals and Systems

Top Courses for Electronics and Communication Engineering (ECE)

Top Courses for Electronics and Communication Engineering (ECE)

Related Content

Mock Test Series

Subject-wise Courses

Subject-wise Courses

Other Engg Branches

Company