**EXAMPLE 9.32. Find the differential entropy H(X) of the uniformly distributed random variable X with the following probability density function (pdf): **

**equation**

** for (i) a = 1, (ii) a = 2, and (iii) a = **** . **

**Solution:** We know that the differential entropy of *X* is given by

**equation**

Making use of given pdf, we have

**equation**

Now, we have

(i) a = 1 , H(X) = log_{2} 1 = 0

(ii) a = 2 , H(X) = log_{2} 2 = 0

(iii) a = , H(X) = log_{2} = – log_{2} = – 1

It may be noted that the differential entropy H(X) is not an absolute measure of information.

EXAMPLE 9.33. The differential entropy of a random variable X is defined by following equation:

**equation**

Find the probability density function f_{X}(x) for which H(X) is maximum.

**Solution:** We know that f* _{X}*(x) must satisfy the following two conditions:

**equation**

**equation**

where μ is the mean of X and σ

^{2}it its variance. Since the problem is the maximization of H(X) under constraints of equations (i) and (ii), therefore, we use the method of Lagrange multipliers as under:

First, we form the function:

**equation**

**equation**

where the parameters l

_{1}, and l2 are the Lagrange multipliers. Then the maximization of H(X) requires that

**equation**

Thus, log

_{2}f

_{X}(x) – log

_{2}e + l

_{1}+ l

_{2}(x-μ)

^{2}

or

**equation**

Hence, we obtain

**equation**

In view of the constraints of equations (i) and (ii), it is required that l

_{2}< 0.

**equation**

Then, equation (v) can be rewritten as

f

_{X}(x) = ae

^{-b2}(x -μ)

^{2}

Substituting equation (vi) into equations (i) and (ii), we get

**equation**

**equation**

Solving equations (vii) and (Viii) for a and b

^{2}, we get

a = and b

^{2}=

Substituting these values in equation (vi), we observe that the desired f

*(x) is given by*

_{X}**equation**

which is the probability density function of Gaussian random variable

*X*of mean μ and variance σ

^{2}.

**EXAMPLE 9.34. Show that the channel capacity of an ideal AWGN channel with infinite bandwidth is given by**

**equation**

**where**

*S*is the average signal power and Ƞ/2 is the power spectral density (psd) of white gaussian noise.*(U.P.S.C. I.E.S. Examination-1999)***Solution:**We know that the noise power

*N*is given by N = ȠB.

DO YOU KNOW? |

A second type of noise, impulse noise, is also encountered in the channel. Impulse noise is characterized by long quiet intervals followed by bursts of high amplitude noise pulses. |

Also, according to Shannon-Hartley law, the channel capacity is given by

C = B log_{2} b/s

In this expression, substituting N = ȠB, we get

C = B log_{2}

Let S/(ȠB) = l. Then, we write

**equation**

Now **Equation**

**equation**

**NOTE:** It may be noted that equation (i) can be used to estimate upper limits on the performance of any practical communication system whose transmission channel can be approximated by the AWGN channel.

**EXAMPLE 9.35. Given an AWGN channel with 4 kHz bandwidth and the noise power spectral density Ƞ/2 = 10 ^{-12} W/Hz. The signal power required at the receiver is 0.1 mW. Calculate the capacity of this channel. **

*(U.P. Tech, Sem. Exam., 2005-2006)***Solution:**Given that B = 4000 Hz

S = 0.1(10

^{-3})W

N = ȠB = 2(10

^{-12})(4000) = 8(10

^{-9}) W

Thus, = = 1.25(10

^{4})

And, by equation (14.50), we have

C = B log

_{2}= 4000 log2 (1 + 1.25 (10

^{4})]

C = 54.44 (10

^{3}) b/s

**Ans.**

**EXAMPLE 9.36. An Analog signal having 4 kHz bandwidth is sampled at 1.25 times the Nyquist rate, and each sample is quantized into one of equally likely levels. Assume that the successive samples are statistically independent.**

**(i) What is the information rate of this source?**

**(ii) Can the output of this source be transmitted without error over an AWGN channel with a bandwidth of 10 kHz and an S/N ratio of 20 dB?**

**(iii) Find the S/N ratio required for error-free transmission for part (i).**

**(iv) Find the bandwidth required for an AWGN channel for error-free transmission of the output of this source if the S/N ratio is 20 dB?**

*(U.P.S.C. I.E.S. Examination, 1999)***Solution:**(i) Here, f

*= 4(10*

_{m}^{3})Hz

We know that Nyquist rate f

*is given by*

_{s}f

*= 2f*

_{s}

_{m}Nyquist rate = 2fm = 8(10

^{3}) samples/s

Also, we have

*r*= 8(10

^{3})(1.25) = 10

^{4}samples/s

Further, we know that entropy is expressed as

**equation**

Here P(x

_{i}) =

Hence, H(X) = log2 256 = 8 bits/sample

The information rate

*R*of the source is given by

R = rH(X) = 10

^{4}(8) b/s = 80 kb/s

**Ans.**

**(ii) Again, we know that channel capacity is given by**

C = B log

_{2}bit/s

Hence, C = B log

_{2}104log

_{2}(1 + 10

^{2}) = 66.6 (10

^{3}) b/s

DO YOU KNOW? |

The output of a discrete information sources is a message that consists of a sequence of symbols. The actual message that is emitted by the source during a message interval is selected at random form a set of possible messages. |

Here, since R > C, error-free transmission is not possible.

(iii) The required S/N ratio can be found by

C = 104 log_{2} ≥ 8 (10^{4}).

or log_{2 } ≥ 8

or ≥ 2^{8} = 256

or ≥ 255 ( = 24.1 dB) **Ans.**

Thus, the required S/N ratio must be greater than or equal to 24.1 dB for error-free transmission

(iv) The required bandwidth *B* can be found by

C = B log_{2} (1 + 100) __>__ 8(10^{4})

or **equation**

and the required bandwidth of the channel must be greater than or equal to 12 kHz. **Ans. **